10:34 Detector Bias a leakage currents all ok. (Attachments 1 and 2)
      Temperatures all ok (Attachment 3)
      All system wide checks ok other than clock status see attachment 4
      DAQ continues to run ok (R124) (No stop at 6:30am)
      DAQ running at about 14MB/s to disc which is faster than before.
      Most likely due to the increased rate in detector 1 which is represented at FEEs 17, 20, 22 and 23 in
      attachment 5.

      Also attached are the spectra for layouts 1-4 & 7-8 as attachments 6-11
      When looking at the pulser spectrum in layouts 3 and 4 (attachments 9 and 10) we see that the pulser peaks
relating to this detector is broader shown by
      the decreased peak height.

      Comparing the waveforms in attachments 11 and 12 to those taken on Monday 31st Oct we see that the noise
in nnaida 5 and 6 has been greatly reduced.
      On the others however there appears to be an increase in the amplitude of the noise.

      11:25 The second putty window for leakage currents was found to have been frozen for the past 24 hours.
Was reloaded and updated leakage currents are in attachment 13 and 14

12.15 Analysis of R124_291 shows ADC & disc data synchronised - see attachment 15
      Analysis of data file - see attachment 16

18:26 DAQ stopped, presumably during LN2 fill cycle.
      Restarted the merger and tried "Stop" on the DAQ control page. ~50% of nnaida now show "error."
      R124 closed 18.29
      Could not regain control of the system. Jorge switched BRIKEN to its internal clock and FEEs power cycled @
19.20.

23.35 DAQ started (R126 - no storage mode)

      DSSSD bias & leakage currents OK - see attachment 

Attached is the updated leakage current spreadsheet (Attachment 1)  as pf 5/11/2016 it is up to date with the elog.

Also added is the recording of ambient temperature. This is recorded in degrees C and plotted relative to 21 degrees C to better show on the same plot as leakage currents. (Screenshot in attachement 2)

Also attached is the spreadsheet in csv formatting.

10.55 DAQ found to have stopped at 06:47 during LN fill cycle.

10:57 Screen shots of system after stop

11.20 R122 started.
      System check screen shots attached at 11:08

18.20 System been running stably all day. Leakage currents attachments 14+15.
      LN2 fill cycle shortly, anticipate DAQ stop.

19.16 DAQ stopped.
      Closed file R122.

19.37 Temp 25.3 DP 6.1 Hum 31.1
      Added 2 extra stacks of lead bricks to protect AIDA. Now ~15cm of lead (Attachment 17)

23.30 DSSSD bias & leakage currents OK - see attachments 18-19
      FEE74 temperatures OK - see attachment 20

23.54 DAQ starts (R124)
      Merge rate 1.8M items/s, c. 13Mb/s to disk c. x2 earlier rates
      Dominated by significantly higher rates from DSSSD#1 
      FEE64 #17 ASIC 2*L pulser peak widths ~300 ch FWHM 
      Needs to be investigated at next entry opportunity

Stack of Pb bricks upstream of AIDA+BRIKEN to protect from incoming beam.

13.35 Swapped HDMI cables from master MACB to MACB #3 & #7
      for HDMI cables from MACB #7 ports 1 & 2 to MACB modules #8 & #9

      ReSYNC and stsrem wide test failures - see attachments 1-5

15.00 Swap HDMI cable from MACB #2 port 1 to MACB #3 next

      ReSYNC and system wide test fails - see attachments 6-10

      *Conclusion* all of HDMI cables between HDMI cable from nnaida6 and master MACB
      have now been swapped for other HDMI cables between MACB # 7 and MACB # 8, 9, 10
      but we still observe ReSYNC and clock issues 

16.05 CG points out that I have misinterpreted Elog 258 - switch back MAC
      mode settings to the following

      MACB #  1  2  3  4  5  6  7  8  9 10 11      spare 1   spare 2
      Mode    5  2  2  3  3  3  3  3  3  3  3
      Rev B      y  y                              50MHz?    y

      FEE64 power cycle

      ReSYNC and system wide fails - see attachments 11-16


16,50 Re-considering options
      Install MACB #11 (rev B) mode 0
      Will attempt to use 25MHz correlation scaler clock NIM output from master MACB to BRIKEN+RIBFDAQ

      Power cycle FEE64s

      ReSYNC OK
      System wide checks OK but some issues remain - see attachments 17-25

17.10 DAQ starts
      Writing to disk file R115, merge rate 0.9M data items/, TapeServer ~7Mb/s

17.20 Analysis of R115_1 confirms ADC data synchronised - see attachment 26

17.25 MERGE stats look OK - see attachments 27-29

17.35 System wide tests SYNC errors & SYNC pulses OK

17.38 DAQ stalled - manual start of ORNL clover LN2 fill cycle by Nishimura-san
      Most recent file R115_5 last accessed 17.38

      DAQ stop OK

      Disable histogram and data transfer #1 
      DAQ RESET
      DAQ SETUP
      Restart merger and TapeServer
      Enable histogram and data transfer #1 

17.47 DAQ start (R116)

17.58 DAQ stopped again at end of fill cycle.
      1. Disable histogram and data transfer #1
      2. DAQ RESET
      3. DAQ SETUP
      4. Restart merger and TapeServer
      5. Enable histogram and data transfer #1 
      Had to repeat 1-5 multiple times to get all links working with the merger. R117 created during this (empty).

18.08 Run R118 started.

18.45 TD changing things for runs R119-R120.

18.50 Run R121 started.
      First few minutes are pulser run 30-10,000 @ 250Hz on coarse dial.

18.54 Beam ON.
      Continuing R121. Pb shield immediately after final MUSIC detector.

21.35 DAQ continues OK
      Most recent file R121_30

      All system-wide tests OK *but* note nnaida5 now fails clock status test in addition to combinations of
nnaida7/8/15/16
      Merge/Tapeserver desktop, Merge statistics 
      - see attachments 30-36

21.45 R121_32 ADC data synchronised

Overnight

Phong Vi completed the pulser walkthrough interrupted earlier this evening.
BNC PB-4 amplitude settings 30,000 - 20,000 @ 10,000 step, 8,000-2,000 @ 2,000 step  
The data is in files R121_62, _63 and _64 (which have been copied to d05 for further analysis).

00.40-02.30 Discussion with PJCS
 
      Confirm both NIM bin voltages OK

      Move master MACB to lower NIM bin (with BNC PB_4, Ortec 433, 2x CAEN N1419B, 1x CAEN N1419)
       successful ReSYNC
       system-wide checks clock test fails from zero to all of the following FEE64s - nnaida7, 8, 15 & 16
       SYNC count test OK *except* nnaida19 which has greater number of SYNC pulses
       online analysis indicates nnaida19 ADC data not synchronised with other FEE64s by ~200ms

      Replace master MACB rev B mode 0 with MACB module #13 mode 5 (external clock 50MHz TTL)
      Move MACB module 2 from upper NIM bin to lower NIM bin
      Swap nnaida19/port1 <-> nnaida20/port2 

      Power cycle FEE64s and restart
      System wide checks fail - see attachments 1-4

      Re-seat HDMI cabling master MACB -> MACB modules 3-6
      System wide checks fail - see attachments 7-10

      Swap ports of 2x HDMI cables from master MACB
      Power cycle FEE64s
      Re_SYNC & system wide checks fail - see attachments 11-15

10.20 ReSYNC fails for all TS_SYNC_PHASE settings 0-3

      DK checks MACB mode settings
      master -> 5
      module #2 2
      module #3 2
      modules #4-11 3

      Check HDMI cable seating nnaida6 to master MACB

      No change - ReSYNC fails, system wide checks fail as before

      Per PJCS e-mail check MACB settings per https://elog.ph.ed.ac.uk/AIDA/258
      and change MACB module # 7 from mode 3 to mode 2. MACB settings are now

      MACB #  1  2  3  4  5  6  7  8  9 10 11      spare 1   spare 2
      Mode    5  2  3  3  3  3  2  3  3  3  3
      Rev B      y  y                              50MHz?    y

      *BUT* remember the system has been in the previous state since Elog 258
      abd has been working OK

      No change - ReSYNC fails, system wide checks fail as before

      Check 50MHz TTL external clock signal and observe transients and width changes
      in clock signal - see attachment 16
      This may not be *the* explanation of current problems but I doubt the the hardware
      will be 'happy' with it either

      JAR cycles NIM signal through NIM-TTL signal converter several times to produce
      better (no transients or width changes) clock signal with c. 3.2V amplitude

      Power cycle and restart FEE64s
      No change - ReSYNC and system wide checks fail - see attachments 17-21

12.40 Re-checked HDMI cabling 
      found master MACB port 1 -> MACB #7 next, port 2 -> MACB #3 next
      change to master MACB port 1 -> MACB #3 next, port 2 -> MACB #7 next
      Power cycle FEE64s

      ReSYNC partial failure (left side OK, right side of tree fails), system wide checks fail - see attachments
22-26

13.05 Swap HDMI cables from master MACB to parents, i.e. HDMI cable master port 1 to MACB #3 next
      changes to master port 2 to MACB #7 next 

      ReSYNC and system wide checks fail - see attachments 30-34
      

Assumes 3-phase ac power (to Julabo FL11006 recirculating chiller) and single-phase ac power
(to AIDA NIM bins, PSUs, network switches, USB-controlled ac mains relay and Raspberry Pi)
is connected and switched on.


All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


1) Start Julabo FL11006 Recirculating Chiller

   See manual (page 19)  http://www2.ph.ed.ac.uk/~td/AIDA/Information/Julabo%20FL11006.pdf 
    
   Set point +20 deg C

   Check flow of coolant by inspecting the two flow indicators which are located on top of
   the AIDA stand

   Do NOT proceed if the Julabo FL11006 Recirculating Chiller is not operating, or if there
   is little, or no, flow in one, or both, flow indicators.

2) Start interface to USB-controlled ac mains relay to AIDA data acquisition servers

   On the AIDA Dell T410 Workstation aidas1 switch to Desktop 7 and start a Firefox web browser
   
   Enter URL   http://nnrpi1:8015/AIDA/Rly16/

   See attachment 1

   Select 'Sequence ALL On' to powerup the AIDA data acquisition servers

   Check all relays Relay 1 to Relay 8 move from the 'Switch ON' to 'Switch OFF' row

   See attachment 2

   Wait 10 minutes (yes - really!) before proceeding

3) Start Httpd Data Acquisition server

   See attachment 3

   Switch to Desktop 1 and select the yellow MIDAS icon 'Httpd for DataAcquisition' which will start a terminal
   to display server messages as follows

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-504.8.1.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS@aidas and MIDAS_LIBRARY = /MIDAS@aidas/TclHttpd/Linux64
PATH =
/MIDAS@aidas/bin_Linux64:/MIDAS@aidas/TclHttpd/Linux64:/homes/npg/root/bin:/homes/npg/root/bin:/homes/npg/root/lib:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "jhi+u8jzdbpl"
httpd started on port 8015

Custom startup from /MIDAS@aidas/config/TclHttpd/aidas1@8015/startup.tcl
DefineMessage defined with server at http://elog.ph.ed.ac.uk:8015/MessageLoggerServer
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
RunControlClient loaded
Completed custom startup from /MIDAS@aidas/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl

   See attachment 4

4) Start web browser for interface to AIDA

   Switch to Desktop 1 and start Firefox web browser

   Enter URL     localhost:8015

   See attachment 5

5) Commence startup of AIDA interface

   Select 'AIDA Experiment Control'

   See attachment 6

6) From AIDA Experiment Control tab 

   Select 'AIDA Hardware Control'

   See attachment 7

   Select 'Data Acquisition Run Control'

7) Reset and setup all data acquisition servers

   From 'Data Acquisition Run Control' tab

   See attachment 8

   Select 'Act On ALL Data Acquisition Servers'

   The web page will update (~20s) to show the status of all of the configured AIDA FEE64
   data acquisition servers which are named nnaida1, nnaida2 ... etc

   See attachment 9

   Select 'Reset'

   Wait for update and check that the state of *all* data acquisition servers is 'reset'

   Select 'Setup'

   Wait for timestamp re-synchronisation pop-up

   See attachments 10-11

   Usually all data acquisition servers (nnaida1, nnaida2 ... etc) will ReSYNC OK and you may proceed to step 8.

   Sometimes one, or more, data acquisition servers report ReSYNC not complete.

   Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'

   See attachments 12-13

   From the 'AIDA Hardware Control' tab select 'Master Timestamp Control'

   From the 'Master Timestamp Control' tab

   Select 'Reload'

   Select FEE64 -> nnaida6

   Select 'Perform ReSYNC'

   Wait for timestamp re-synchronisation pop-up.
   All data acquisition servers 9Nnaida1, nnaoida2 .. etc) should ReSYNC OK - if they do not
   please seek expert help

8) Start ASIC controls

   Re-select the 'AIDA Experiment Control' tab and select 'AIDA Hardware Control'

   See attachments 12-13

   Select 'ASIC Control'

   See attachment 14

   From 'ASIC Control' tab
     select 'Act on ALL FEE64 modules' and await page update
     select 'Act on ALL ASICs' and await page update

   See attachment 15

9) Check FEE64 operating temperatures

   Re-select the 'AIDA Hardware Control' tab and select 'FEE Temperatures'

   See attachment 16

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C

   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0
   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows
   
   Desktop 6 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off' and *now* seek expert help

10) Perform system-wide diagnostic checks

    Re-select the 'AIDA Hardware Control' tab and select 'System Wide Checks'

    See attachments 17-18

11) Start servers for data merging and output to disk

    Switch to Desktop 2 and start the 'Httpd for Merge/TapeServer', 'Merger' and 'TapeServer' servers
    by selecting the three adjacent MIDAS in sequence

    See attachment 19

    Three terminal sessions will pop-up to display server messages. These terminals should be moved to
    left of the desktop and arranged as shown 

    See attachment 20

    This will enable you to simultaneously observe the latest messages from each server and the activity
    spinners of the Merger and TapeServer servers 

12) Start web interface to Merger and TapeServer

    Start Firefox web browser

    Enter URL     localhost:8115

    See attachment 21

13) Open data file

    Select 'Tape Service'

    See attachment 22

    In the 'TapeServer Service' tab complete the Volume Name, Run Name and Run Number fields.
    Typically the volume name will the experiment ID (e.g. NP1234) and Run Name (R). 

    See attachment 23

    Select 'Go' 

    See attachment 24

    This will create an experiment directory if it does not already exist (e.g. /TapeData/NP1234)
    and will open a data file within this directory with a filename concatenating the Run Name
    and Run Number fields - if the selected file already exists a new filename (Run Number) will
    be suggested

14) Setup merger

    Re-select the 'AIDA' tab and select 'MERGE Control'

    See attachment 25

    Select 'Setup & Configure'

    See attachment 26

    Select 'Go'

    See attachment 27

    Select 'Toggle Merge Program Pause State' 

    See attachment 28

15) Enable histogramming and data transfer #1 links

    Switch to Desktop 1 and re-select 'Data Acquisition Run Control' tab

    Select Histogramming 'Enable All' and Data Transfer #1 'Enable All'

16) AIDA is now ready to commence collecting data 
    - you can proceed from step 5 of https://elog.ph.ed.ac.uk/AIDA/412

10.20 DAQ found stalled

      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
      AIDA SYNC stats - see attachment 4 - stats 10.1M/381/3600 => DAQ stalled at 23.05 + 7.4h => c.06.29
      System wide checks OK *except* SYNC pulses & SYNC errors - see attachments 5-8

      Conclude DAQ stalling at c. 06.30/18.30 due to start of ORNL LN2 dewar refill cycle
      but unrelated to either the BRIKEN-supplied 50MHz TTL clock signal to external clock
      input of Master MACB or the ground connection provided by this coaxial cable.

      Which implies the problem is either EMI/RFI or ac mains electrical ground. 

      AIDA powered by two single phase outlets from one of four sections of a single phase ac mains distribution
      box with power supplied by two heavy duty (3cm dia) cables from Denkenseiki Noisecut AVR type PMN (AVR4, B3F
      104V/51A). AIDA is connected to both NEMA 5-15R-style outlets of section A, LN2 refill to section D, BRIKEN
      to sections B, C and D.

      Ambient temperature +24.8 deg C, d.p. +4.1 deg C, RH 26.3%

15.00 DSSSD cabling drain wire attachment to AIDA adaptor PCB (FEE64 # 5 & 6) not tightly secured.
      CG re-secured drain wire and this has significantly reduced 100kHz period transient noise
      as observed at preamplifier outputs.

14.48 Restart DAQ (R110)
      Master MACB mode 5, 50MHz TTL clock to external clock input

14.57 DAQ stalled

      All system wide checks OK *except* SYNC errors - see attachment 9
      ASIC clock timestamp - nnaida19 fails 
      N.B. SYNC pulses check OK, counters incrementing between successive tests
      The latter is a problem we have seen previously but has not stopped DAQ running OK

aidas1> ./analyser /TapeData/Sept2016/R110_2
 *** TDR format 3.3.0 analyser - TD - October 2016
 *** ERROR: READ I/O error:       5002
                   blocks:      11861
          ADC data format:   82556471 (  877427.0 Hz)
        Other data format:   14217428 (  151105.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         27 (       0.3 Hz)
                               RESUME:         27 (       0.3 Hz)
                              SYNC100:     861083 (    9151.8 Hz)
                           FEE64 disc:   13353468 (  141923.4 Hz)
                             MBS info:       2823 (      30.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       1976 (      21.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       94.089 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.084                0.000
  2                0.076                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.109                0.000
  6                0.012                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.003                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.032                0.000
 16                0.013                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.125                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.142                0.149
 24                0.076                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100       Disc        MBS      Other  
HEC Data
  0          0          0          0          0          0          0          0          0          0          0  
       0
  1     942953    5806860          0          0          3          3      35855    5770999          0          0  
      42
  2    4036003    4268394          0          0          2          2      35861    4232529          0          0  
       0
  3    3038422      39132          0          0          0          0      35892       3240          0          0  
       0
  4    1192440      42063          0          0          0          0      35889       6174          0          0  
      68
  5    5685966      39310          0          0          5          5      35849       3451          0          0  
       0
  6    3670394      46605          0          0          2          2      35885       7893       2823          0  
      70
  7    2661022      43508          0          0          0          0      35890       7618          0          0  
     231
  8    2165335      41382          0          0          0          0      35890       5492          0          0  
     359
  9    3037520     863964          0          0          1          1      35889     828073          0          0  
       0
 10    2616647     179334          0          0          0          0      35889     143445          0          0  
       0
 11    1083633      39064          0          0          0          0      35889       3175          0          0  
       0
 12    1759629      38368          0          0          0          0      35888       2480          0          0  
       0
 13    2419715     280498          0          0          0          0      35891     244607          0          0  
       0
 14    3189205    1325559          0          0          0          0      35887    1289672          0          0  
     747
 15    4886680      39482          0          0          2          2      35871       3607          0          0  
       7
 16    5087977      38869          0          0          1          1      35886       2981          0          0  
       0
 17    2905638     329411          0          0          0          0      35888     293523          0          0  
       0
 18    3327386      46016          0          0          0          0      35888      10128          0          0  
       0
 19    6770177      40711          0          0          4          4      35842       4861          0          0  
       7
 20    2225755      38524          0          0          0          0      35891       2633          0          0  
     344
 21    2061728      40299          0          0          0          0      35887       4412          0          0  
      90
 22    3052329     512109          0          0          0          0      35890     476219          0          0  
       7
 23    9325689      38673          0          0          3          3      35834       2833          0          0  
       3
 24    5414228      39293          0          0          4          4      35862       3423          0          0  
       1
 25          0          0          0          0          0          0          0          0          0          0  
       0
 26          0          0          0          0          0          0          0          0          0          0  
       0
 27          0          0          0          0          0          0          0          0          0          0  
       0
 28          0          0          0          0          0          0          0          0          0          0  
       0
 29          0          0          0          0          0          0          0          0          0          0  
       0
 30          0          0          0          0          0          0          0          0          0          0  
       0
 31          0          0          0          0          0          0          0          0          0          0  
       0
 32          0          0          0          0          0          0          0          0          0          0  
       0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0

 *** Program elapsed time:    1.500s ( 7907.333 blocks/s, 494.208 Mb/s)

15.48 Restart aidas1
      Power cycle FEE64s

      Restart AIDA DAQ

      Same problem - DAQ stall (no output from merger) at, or shortly after, Start ASIC Readout with
      multiple FEE64s with zero AIDA SYNC rate
      *All* system wide checks OK

      Cannot recover DAQ control of FEE64s with merger restart - power cycle FEE64s again


23.58 Attempt to change master MACB from MACB module #11 mode 5 to MACB module #13 mode 0

      Successive fuse failures of USB ac mains relay

      System wide ASIC clock timestamp check nnaida 5/6/7/8 13/14/15/16 21/22/23/24 i.e. first half of MACB tree

      Re-seat HDMI cabling from  master MACB root module to first half of MACB tree - ReSYNC fails - see attachments
10 & 11

   

08.56 DAQ continues OK

      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
      System wide checks OK *except* SYNC pulses - see attachments 4-6

      Most recent file R101_249 ADC & disc data synchronised - see attachments 7-8
      spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)
      spectra 1-32 and 100-132 per FEE64 data items as function of timestamp (20us/channel)

09.03 DAQ stopped


17.10 DAQ started (R103) no storage mode
      Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel.

18.55 Checked DAQ status and it has stopped. No file write times so can't easily check exactly when (see SYNC
stat page to calculate).
      All system checks passed except SYNC errors (see attachments 9-12)
      Upon attempting to Stop the DAQ, could not contact nnaida11.

      AIDA SYNC statistics c. 1.76M SYNCs per FEE64. SYNC rate = 381Hz => elapsed time c. 1.3h => DAQ stalled c.
18.30

23.05 DAQ started (R104) no storage mode
      Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel.

11.24 DAQ found stalled
      Most recent file R95_168 last accessed 06.31

      R95_168 ADC & disc data synchronised - see attachment 8
      spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)

aidas1> ./analyser /TapeData/Sept2016/R95_168
 *** TDR format 3.3.0 analyser - TD - October 2016
 *** ERROR: READ I/O error:       5002
                   blocks:      14568
          ADC data format:  108169612 ( 1126585.0 Hz)
        Other data format:   10690700 (  111343.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         31 (       0.3 Hz)
                               RESUME:         30 (       0.3 Hz)
                              SYNC100:     877566 (    9139.8 Hz)
                           FEE64 disc:    9810241 (  102173.5 Hz)
                             MBS info:       2832 (      29.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       2160 (      22.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       96.015 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                1.994                2.016
  6                1.408                1.439
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.015                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.045                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.099                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100       Disc        MBS     
Other   HEC Data
  0          0          0          0          0          0          0          0          0          0         
0          0
  1     350112      43200          0          0          0          0      36623       6577          0         
0         48
  2    1247240      43778          0          0          0          0      36622       7156          0         
0          0
  3    2547061      39987          0          0          0          0      36626       3361          0         
0          0
  4     881007      40879          0          0          0          0      36623       4256          0         
0          5
  5   23965264    2536358          0          0         13         12      35831    2500502          0         
0         38
  6   23702370    1860427          0          0         11         11      36086    1821487       2832         
0        844
  7    1895595      40743          0          0          0          0      36623       4120          0         
0          1
  8    1603094      41634          0          0          0          0      36626       5008          0         
0        325
  9    3180182    1849501          0          0          0          0      36625    1812876          0         
0          0
 10    2576178      97708          0          0          0          0      36622      61086          0         
0          0
 11     447806      39812          0          0          0          0      36625       3187          0         
0          0
 12     905803      39001          0          0          0          0      36618       2383          0         
0          0
 13    2226595     147679          0          0          0          0      36624     111055          0         
0          0
 14    3021228    2593022          0          0          0          0      36626    2556396          0         
0        513
 15    4146089      40401          0          0          2          2      36617       3780          0         
0          0
 16    4264696      39555          0          0          0          0      36619       2936          0         
0          0
 17    2934508     416504          0          0          0          0      36623     379881          0         
0          0
 18    3319005      56583          0          0          0          0      36619      19964          0         
0          0
 19    6346836      41482          0          0          2          2      36604       4874          0         
0          1
 20    2392653      39249          0          0          0          0      36623       2626          0         
0        283
 21    2056421      41512          0          0          0          0      36622       4890          0         
0         96
 22    2824822     522339          0          0          0          0      36624     485715          0         
0          5
 23    7402741      39352          0          0          3          3      36590       2756          0         
0          1
 24    3932306      39994          0          0          0          0      36625       3369          0         
0          0
 25          0          0          0          0          0          0          0          0          0         
0          0
 26          0          0          0          0          0          0          0          0          0         
0          0
 27          0          0          0          0          0          0          0          0          0         
0          0
 28          0          0          0          0          0          0          0          0          0         
0          0
 29          0          0          0          0          0          0          0          0          0         
0          0
 30          0          0          0          0          0          0          0          0          0         
0          0
 31          0          0          0          0          0          0          0          0          0         
0          0
 32          0          0          0          0          0          0          0          0          0         
0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0

 *** Program elapsed time:    2.449s ( 5948.019 blocks/s, 371.751 Mb/s)


      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
      System wide checks OK *except* SYNC pulses & errors - see attachments 4-7

      Restart merger
      DAQ STOP
      System wide checks 
      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

11.46 Start DAQ (R96)

16.28 Stop DAQ OK

16.30 Start DAQ (R97)
      BNC PB-4 pulser OFF (background run)

17.04 Stop DAQ

17.08 Ambient temperature +25.0 deg C, d.p. +8.4 deg C, RH 34.8%

17.11 Start DAQ (R98)
      BNC PB-4 pulser ON
      60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3

17.30 DAQ stalled due to excessive diak activity from data copying & analysis tasks
      Power cycle required

18.09 DAQ start (R99)

18.38 DAQ stalled 
      Most recent file R99_7 last accessed 18.38
      Power cycle rquired

19.42 DAQ start (R101)
      BNC PB-4 pulser ON
      60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3
(Phong's correction: 60Co source position was in front of AIDA snout toward upstream)
      ASIC settings 2016Nov01-18.02.17

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 7 - 2x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 3 = DSSSD # 4

   Addr 01

   Ch 0 = DSSSD # 5
   Ch 1 = DSSSD # 6

   Sequentially select each terminal and obtain a screenshot (Alt-PrtScn) of DSSSD #1 = #6 detector
   bias voltages and leakage currents
   Upload screenshots to BRIKEN Elog

   If B2F/F11 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 6 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

5) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~0.5-2M data items/s) consistent with value observed at start of run

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

6) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


How to Stop Run

1) Workspace 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check number of blocks written to disk is increasing

   See attachment 1

   If TapeServer data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303

2) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~0.5-2M data items/s)

   See attachment 2

   If MERGER data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303

3) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

   Stop DAQ - select red 'STOP' button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Check state of *all* data acquisition servers is 'stopped'
   Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags

   See attachment 3

   N.B. Occasionally the state of all data acquisition servers does not change to 'stopped'
        and if this happens simply re-try by re-selecting 'STOP' 

4) Workspace 2 - Firefox web browser - MERGE tab

   Pause MERGE input - select 'Toggle Merge Program Pause State'

   See attachment 4

5) Workspace 2 - Firefox web browser - TapeServer tab

   Stop TapeServer and close disk file - select red 'STOP' button

   See attachment 5

6) Complete BRIKEN Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key
    additional comments
    your name

    Complete run sheet for written logbook

All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!



How To Start Run 

1) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check DAQ stopped (green 'GO' button displayed)

   Check state of *all* data acquisition servers is 'stopped'
   Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags

   See attachment 1

2) Workspace 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check TapeServer stopped and disk file closed (green GO button displayed)

   See attachment 2

3) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check Merge Program paused - see banner 

   Merge State = GOing     :    paused     :    xfer enabled     :    output paused

   If not paused select 'Toggle Merge Program Pause State'

   See attachment 3

4) Workspace 1 - Firefox web browser - ASIC Control tab

   Select 'Reload'

   Check 'Act on ALL FEE64 modules?' selected

   Check 'Act on ALL ASICs' selected

   Select 'Select required function' -> 'Check ASIC Control'

   After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
   Check all ASICs are OK and status = 0x001F0000.
   Dismiss pop-up by selecting OK

   See attachments 4-5

5) Workspace 1 - Firefox web browser - Run Control tab 

   Start DAQ - select green GO button

   Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates 

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags

   See attachment 6

6) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check *all* enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are inactive call for expert help and do NOT proceed!

   Observe active spinner in MERGER server terminal window

   See attachment 7

7) Workspace 2 - Firefox web browser - TapeServer tab

   Check 'Volume Name' and 'File Name'

   Start TapeServer and open disk file - select green 'GO' button

   See attachment 8

8) Workspace 2 - Firefox web browser - MERGE tab

   Select 'Reload' several times and check all enabled data links (0-23) are always active
         bright green = active, olive green = inactive, red = disabled 

   N.B. If some data links are *always* inactive call for expert help and do NOT proceed!

   Select 'Toggle Merge Program Pause State' to enable data merging 

   Select 'Reload' to observe merge rate (~ 9000 data items/s)

   Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser

   See attachment 9

9) Workspace 1 - Firefox web browser - Run Control tab

   Select 'Start ASIC Data Readout'

   See attachment 10

10) Workspace 2 - Firefox web browser - MERGE tab

    Select Reload several times and check all enabled data links (0-23) are active
         bright green = active, olive green = inactive, red = disabled 

    N.B. If some data links are *never* active call for expert help and do NOT proceed!

    Select 'Reload' to observe merge rate (~ 0.5-2M data items/s)

    See attachment 11

11) Workspace 2 - Firefox web browser - TapeServer tab

    Select 'Reload' and observe number of blocks written to disk increasing

    See attachment 12

12) Workspace 1 - Firefox web browser - Control tab

    Select 'Save/Restore Settings'

    From 'Save/Restore Module Settings' tab

     select 'Reload'

     select 'Module Names' whose settings should be saved (usually all)

     select 'Generate new DB key' and make a note of the DB key generated
          the DB key will have the format YYYYMMMDD-HH.MM.SS

          The DB key will be used to name a directory to store AIDA settings
          and is created uid: npg, gid: npgstaff, perms 755. However, its
          contents are written uid: nfsnobody, gid nfsnobody. The default
          directory (DB Key) permissions do not work - change the permissions
          to 777 - before attempting to save data to this directory (DB Key).

     launch a terminal and change the permissions of the DB key directory using the command

     chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]

     select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
     the directory (DB key)

11) Update BRIKEN Elog with the following information

    AIDA
    run filename (e.g. R99)
    [# data blocks]
    start date & time
    [stop date & time]
    ASIC settings DB key 
    additional comments
    your name

    Complete run sheet for written logbook

11.00 DAQ continues OK
      DAQ STOP OK
      Most recent files R91_568 last accessed 10.59

      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
      System wide checks OK *except* SYNC pulses & errors - see attachments 4-7

14.20 Per FEE64 screenshots p+n junction and n+n ohmic strip waveforms - see attachments 8-11
      Preamplifier output noise looks OK for all FEE64s *except* nnaida5 & nnaida6 (which
      have significantly higher ADC and disc rates than other FEE64s)

21.10 DAQ found stalled
      Most recent file R94_99 last accessed 18.35

      Restart merger
      DAQ STOP
      System wide checks 
      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

21.16 DAQ start (R96)

The following spectrum layouts have been created

1  All Rate spectra
2  All HitRate Spectra
3  p+n junction strip 1.8.L spectra
4  n+n ohmic strip 1.8.L spectra
5  p+n junction strip 1.8.H spectra
6  n+n ohmic strip 1.8.H spectra
5  p+n junction strip 1.8.W waveforms
6  n+n ohmic strip 1.8.W waveforms

The layouts can be found in the directory ~/LayOut and should be copied to /tmp

N.B Currently you cannot restore a layout to a spectrum display with > 16 galleries.
    If spectrum display uses > 16 galleries, switch spectrum display to 16 galleries
    or less before restoring a spectrum layout.

Cannot restore layout if currently displaying >16 galleries

Obtain the error message 'loading Spectrum.tcl > can't read "dbtagx(31,1)": no such element in array'

10.53 DAQ found stalled 
      Most recent files R89_102 last accessed 07.17

      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3

      All system wide checks OK *except* Clock status, SYNC errors & SYNC pulses - see attachments 4-8 - unusual
messages/reports

      One merger error messages - link 18 - see attachment 9

      Restart merger
      DAQ STOP OK
      DAQ RESET & SETUP OK
      ReSYNC OK
      All system-wide checks OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

      Standard soft reset did not work - one or more FEE64s dropped data transfer #1 link at DAQ GO

      Retry - one, more FEE64s indicate error state - run control tab unresponsive 

      Power cycle FEE64s 

Following analysis of AIDA ADC & disc readout ( see https://elog.ph.ed.ac.uk/AIDA/404 )
changed timestamp resolution from 1us/channel to 20.48us/channel


Offline sort (from disk file)

Attachments 1-3  s1-s32  # ADC data items per FEE64, s100-132  # disc data items per FEE64 (20.48us/channel)

Attachments 4-5  s1000   4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)

Attachments 6-7  MIDASsort terminal session logs for *offline* sort


Online sort (from TapeServer)

Attachment 8     s1-s32  # ADC data items per FEE64, s100-132  # disc data items per FEE64 (20.48us/channel)

Attachments 9-10 s1000   4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)

Attachments 11-12 MIDASsort terminal session logs for *online* sort

Attachment 13   MIDASsort FORTRAN sort routine sync.f

Attachment 14   Spectrum titles for the above sort routine

Note

1) data is discontinuous as a function of timestamp because we are sampling buffers
from the TapeServer rather than sorting all data as above
2) a low rate of timewarps is reported which appears to be an artifact of the buffer
sampling - no timewarps reported by analyses of the corresponding data files
 

11.30 DAQ continues (R87)
      System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 1-4

11.36 DSSSD bias & leakage currents OK - see attachments 5-8
      FEE64 temperatures OK - see attachment 9

16.46 DAQ continues (R87)
      System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 5-8

18.55 DAQ stalled
      Most recent file R87_274 last accessed 18.47

      Restart merger
      DAQ STOP
      System wide checks 
      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

19.10 DAQ starts (R89)

14.12 DAQ stalled as expected
      Most recent file R86_245 last accessed 06.26

      DSSSD bias & leakage current OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3
     
      Some merger errors messages - all link 4 - see attachment 4
  
      All system wide checks OK *except SYNC errors & SYNC pulses - see attachments 5-8

      Restart merger
      DAQ STOP OK
      DAQ RESET & SETUP OK
      ReSYNC OK
      All system-wide checks OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

14.34 DAQ started (R87)

19.50 DAQ continues (R87)

      System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 9-12