00:05 BigRIPS and BRIKEN stop their daqs
      AIDA currently on file R1_312

00:06 BRIKEN start run 23
      BigRIPS start run 1024
      AIDA on file R1_312
     
      System wide checks all ok

01:05 BigRIPS and BRIKEN stop their daqs
      AIDA on file R1_327

01:06 BigRips Start run 1025
      BRIKEN start run 24
      AIDA file R1_327

      System wide checks all ok

02:02 BigRIPS and BRIKEN stop their daqs
      AIDA on file R1_341
      ASIC load check ok

02:03 BigRIPS start run 1026
      BRIKEN start run 25
      AIDA ON FILE R1_341

02:11 System wide checks all ok
      Leakages and bias ok (Leakages rising again though) - attachment 1
      FEE temperatures ok - attachment 2
      Good event statistics - attachment 3
      Merger and tape server still running - attachment 4

02:23 Beam is off. Will recover soon

02:26 Beam is back

02:27 beam stopped again

02:28 beam back

02:37 Since beam has come back we appear to have lost all implant channels.
      only seeing events in DSSD2 

03:30 ASIC load issue resolved. AIDA data should be good from R1_353

      Beam continues to turn off and on

03:52 Since ~2am beam has been passing through the dssd
      Implantation spectrum - attachment 5
      Upon investigation the fixed degrader had fallen on the left and was at a 30 degree angle.
      It has now been put back in position
      AIDA file when fixed R1_360
      BRIKEN reset the timestamp
      Whilst sorting the ASIC issue bigRips moved onto run 1029
      BRIKEN run 1028. Full details in their elog

03:55 briken and bigRIPS stop runs
      bigRIPS start run 1028
      BRIKEN run 27
      AIDA file R1_360

04:29 Implant hit spectra after the degrader was replaced - attachment 6

04:58 BigRIPS and BRIKEN stop daqs
      AIDA on file R1_376

04:59 BigRIPS starts file 1029
      BRIKEN starts run 28
      AIDA at very end of file R1_376

05:20 System wide checks ok
      bias and leakage currents ok - attachment 7

05:26 Beam still dropping out for short moments of time

06:03 BigRips and BRIKEN stop their runs
      AIDA on file R1_393

06:04 BRIKEN start run 29
      BigRIPS start run 1030
      AIDA on file R1_393

06:28 Beam lost

06:31 Beam returned

06:46 System wide checks ok
      FEE temperatures ok - attachment 8

07:01 BigRIPS and BRIKEN stop their runs
      AIDA on file R1_407

07:02 BRIKEN start run 30
      BigRips run 1031
      AIDA on file R1_407

07:31 system wide checks all ok

07:48 Bias and leakage currents ok - attachment 9

08:01 Briken and BIgRips End current runs
      AIDA file R1_422 E silicon detector has saturation problem (see the attachment). We lower the
gain in round factor of 2. 

16.00   Still no beam  
        Current run /RIBF127R1/R1_197
        Found dropped channels on nnaida1, ASIC4 - see attach 1
        Check ASIC control x2
        No effects
        
17.08   FEE's temperature normal
        statistics normal
        system wide checks ok 
        leakage current and bias ok
        merger and tapeserver ok

        still no beam

17.20   Beam is back

        AIDA RIBF127R1/R1_216
        DAQ start BRIKEN 16
        DAQ start BigRIPS 1016

17.25   Almost all implants in DSSSD#6 - see attach 2
        Beam stopped on AIDA run RIBF127R1/R1_217
        Increasing degrader thickness.

17.30   New degrader: 11m Al plate
        Zero'd all histograms
        Deleted online SortSas

        AIDA RIBF127R1/R1_21
        DAQ start BRIKEN 17
        DAQ start BigRIPS 1017

17.44   Most implants still in #5,6. Adjusted degrader: 14mm Al
        
        Implants now in dets #4,5 preferentially.
        Zero'd all histograms
        Deleted online SortSas

        AIDA RIBF127R1/R1_221
        DAQ continues running in BRIKEN 17
        DAQ start BigRIPS 1018    
       
18.05   Beam lost on AIDA run RIBF127R1/R1_226

18.35   Beam is back

        Zero'd all histograms
        Deleted online SortSas
        
        AIDA RIBF127R1/R1_233
        DAQ start BRIKEN 18
        DAQ start BigRIPS 1019 

19:10   DSSD Rates from R1_239
 *** DSSSD # 1 count:    164076 old count:    109857 dt:      1.02 s  LEC rate:  53189.47 Hz
 *** DSSSD # 2 count:    211989 old count:    140492 dt:      1.02 s  LEC rate:  70139.39 Hz
 *** DSSSD # 3 count:     76697 old count:     49676 dt:      1.02 s  LEC rate:  26507.92 Hz
 *** DSSSD # 4 count:     72050 old count:     47860 dt:      1.02 s  LEC rate:  23730.67 Hz
 *** DSSSD # 5 count:     27482 old count:     17785 dt:      1.02 s  LEC rate:   9512.87 Hz
 *** DSSSD # 6 count:     51983 old count:     34674 dt:      1.02 s  LEC rate:  16980.33 Hz
 *** DSSSD # 1 count:        37 old count:        21 dt:      1.02 s  HEC rate:     15.70 Hz
 *** DSSSD # 2 count:        89 old count:        59 dt:      1.02 s  HEC rate:     29.43 Hz
 *** DSSSD # 3 count:       124 old count:        83 dt:      1.02 s  HEC rate:     40.22 Hz
 *** DSSSD # 4 count:       104 old count:        66 dt:      1.02 s  HEC rate:     37.28 Hz
 *** DSSSD # 5 count:        75 old count:        47 dt:      1.02 s  HEC rate:     27.47 Hz
 *** DSSSD # 6 count:        45 old count:        33 dt:      1.02 s  HEC rate:     11.77 Hz

19:23   FEE's temperature normal
        statistics normal
        system wide checks ok 
        leakage current and bias ok see attachment 4
        merger and tapeserver ok
        current AIDA file R1_244
   
20:01   BigRIPS and BRIKEN stopping the run
        AIDA RIBF127R1/R1_253

20:08   BigRIPS and BRIKEN start
        DAQ start BRIKEN 19
        DAQ start BigRIPS 1020
        AIDA RIBF127R1/R1_254

20:19  FEE's temperature normal
        statistics normal
        system wide checks ok 
        leakage current and bias ok see attachment 5
        merger and tapeserver ok
        current AIDA file R1_258

20.51   Beam and DAQ continue OK
        HEC x vs. y look OK - see attach 6  

21:03   BigRIPS and BRIKEN stopping the run
        AIDA RIBF127R1/R1_268

21.05   Beam lost while changing runs

21.07   Beam is back.
        DAQ start BRIKEN 20
        DAQ start BigRIPS 1021
        AIDA RIBF127R1/R1_270

22:01   FEE's temperature normal
        statistics normal
        system wide checks ok 
        leakage current and bias ok
        merger and tapeserver ok
        current AIDA file R1_282

22:02   BigRIPS and BRIKEN stopping the run
        AIDA RIBF127R1/R1_282

22:03   BigRIPS and BRIKEN start
        DAQ start BRIKEN 21
        DAQ start BigRIPS 1022
        AIDA RIBF127R1/R1_283

23:02   BigRIPS and BRIKEN stopping the run
        AIDA RIBF127R1/R1_296

23:04   BigRIPS and BRIKEN start
        DAQ start BRIKEN 22
        DAQ start BigRIPS 1023
        AIDA RIBF127R1/R1_297

23:40   FEE's temperature normal
        statistics normal
        system wide checks ok 
        leakage current and bias ok see attchment 7
        merger and tapeserver ok
        current AIDA file R1_306

08.58 DAQ continues OK 
      File RIBF127R1/R1_99
      BigRIPS run 1009
      BRIKEN run 9

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39

08.59 *online* analysis - ADC & disc data synchronised - see attachment 1
      analysis of R1_99 OK - see attachment 2 - deadtime <0.5%

09.11 BigRIPS & BRIKEN DAQ stop

      
09.13 AIDA RIBF127R1/R1_103
      BRIKEN 10
      BigRIPS 1010

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - in
           1mm Al - out

09.30 Detector biases; leakage currents OK - see attachments 3 & 4
      FEE64 Temperatures OK - see attachment 5
      good events; statistics OK - see attachment 6
      No warnings/errors reported by merger since DAQ GO - see attachment 7

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

09.50 layout 1-8 spectra - see attachments 8-17
      Note pulser amplitude drifting for + polarity

      *all* spectra zero'd

10.00 BigRIPS & BRIKEN DAQ stop
      AIDA RIBF127R1/R1_114

      Check ASIC control x2

10.03 AIDA RIBF127R1/R1_115
      DAQ start BRIKEN 11
      DAQ start BigRIPS 1011

11.12 Changing degraders (16 mm Al plate check?)
      E17/F11 ambient temperature +24.7 deg C, RH 38.7%, d.p. +9.7 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 70%)
      HEC plot attached - see attach 8 

11.17 AIDA RIBF127R1/R1_132
      DAQ start BRIKEN 12
      DAQ start BigRIPS 1012
      Attempted to start new run. Lost beam. BigRIPS re-optimising

11.40 Found beam

      Zero'd all histograms 
      Cleared online sorting histograms

      AIDA RIBF127R1/R1_139
      DAQ start BRIKEN 13
      DAQ start BigRIPS 1013

12.14 Machine is down on AIDA run RIBF127R1/R1_145
      HEC plot attached - see attach 9
      Implants moved one detector upstream

12.24 Check ASIC Control x2

12.30 Beam back
      
12.31 AIDA RIBF127R1/R1_149
      DAQ start BRIKEN 14
      DAQ start BigRIPS 1014

12.35 Detector biases; leakage currents OK - see attachments 22 & 23
      *online* analysis - ADC & disc data synchronised
      analysis of R1_149 OK - see attachment 24 - deadtime <0.5%

12.39 Beam off RIBF127R1/R1_151
      SRC cooling issues

13.55 Background run
      AIDA RIBF127R1/R1_170
      DAQ start BRIKEN 15, BigRIPS 1015

14.01 Beam still off

14.20 Stop background run
      AIDA RIBF127R1/R1_175

15.30 Beam still off
      DAQ continues OK
      FEE64 temperatures OK - attach 24
      Statistics OK - attach 25
      Leakage currents OK - attach 26
      No merger errors. No nnaida panics from USB console terminal.

      

00:02 Bias and leakage currents ok - attachment one
      System wide check all ok
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      ASIC load check all ok
      Merger still running ok - attachment 4

00:15 Current AIDA file R34_126

00:24 Pb Wall is removed
      Variable degraders in place Current configuration
           1mm Al - Out
           1mm Al - Out
           1mm Al - Out

      Fixed Degrader in place 11m Al (8mm +3mm Plates)

00:40 Variable degraders were out of alignment.
      Have now been realigned.

01:06 Fixed degrader moved closer to F11 plastic.
      Moved from far side of Pb wall to side closest to F11 plastic.
      Fixed degrader thickness increased to 13mm Al (8mm + 3mm + 2mm plates)

      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - Out
           1mm Al - Out
           1mm Al - Out

01:36 Implant hit pattern for current Ni setting - attachment 5
      Looks as if we have a broken bond wire in DSSD6 FEE 8 (Missing horizontal band)
      Also maybe one in DSSD3 FEE 14 (Missing vertical band)

01:54 Degraders changed
      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - in
           1mm Al - in
      Implant Hit spectra - attachment 6

01:59 Degraders changed
      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - out
           1mm Al - out
      Implant Hit spectra - attachment 7

02:08 Start AIDA file R1_0 in directory RIBF127R1
      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39

      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - out
           1mm Al - out
      
02:11 Bias and leakages all ok - attachment 8#

02:26 Briken and BigRIPS start new runs
      Briken run 3
      BigRIPS run 1003
      Current AIDA file R1_4

02:42 MIDAS sort crashed, restarted and sorted R1_7 to get the implant rates:
 *** DSSSD # 1 count:    428279 old count:    369602 dt:      1.02 s  LEC rate:  57547.23 Hz
 *** DSSSD # 2 count:    523171 old count:    452321 dt:      1.02 s  LEC rate:  69485.85 Hz
 *** DSSSD # 3 count:    194416 old count:    168097 dt:      1.02 s  LEC rate:  25812.25 Hz
 *** DSSSD # 4 count:    181257 old count:    156583 dt:      1.02 s  LEC rate:  24198.93 Hz
 *** DSSSD # 5 count:     73675 old count:     63323 dt:      1.02 s  LEC rate:  10152.68 Hz
 *** DSSSD # 6 count:    130418 old count:    112563 dt:      1.02 s  LEC rate:  17511.22 Hz
 *** DSSSD # 1 count:        63 old count:        54 dt:      1.02 s  HEC rate:      8.83 Hz
 *** DSSSD # 2 count:       156 old count:       133 dt:      1.02 s  HEC rate:     22.56 Hz
 *** DSSSD # 3 count:       199 old count:       178 dt:      1.02 s  HEC rate:     20.60 Hz
 *** DSSSD # 4 count:       196 old count:       174 dt:      1.02 s  HEC rate:     21.58 Hz
 *** DSSSD # 5 count:       180 old count:       152 dt:      1.02 s  HEC rate:     27.46 Hz
 *** DSSSD # 6 count:        87 old count:        75 dt:      1.02 s  HEC rate:     11.77 Hz

02:59 BigRIPS and BRIKEN stopping the run
      Ni being implanted into detector 6 so will be increasing degrader
      AIDA current run R1_13

      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - in
           1mm Al - out

03:03 BRIKEN start run 4
      BigRIPS start run 1004

03:16 Bias and leakage currents ok - attachment 9
      System wide checks all ok

03:23 Sorted file R1_16 - Current rates are:
 *** DSSSD # 1 count:    191573 old count:    124591 dt:      1.04 s  LEC rate:  64370.77 Hz
 *** DSSSD # 2 count:    232369 old count:    148461 dt:      1.04 s  LEC rate:  80636.92 Hz
 *** DSSSD # 3 count:     84277 old count:     53763 dt:      1.04 s  LEC rate:  29324.44 Hz
 *** DSSSD # 4 count:     78592 old count:     51020 dt:      1.04 s  LEC rate:  26497.13 Hz
 *** DSSSD # 5 count:     30254 old count:     19685 dt:      1.04 s  LEC rate:  10156.98 Hz
 *** DSSSD # 6 count:     55682 old count:     36607 dt:      1.04 s  LEC rate:  18331.38 Hz
 *** DSSSD # 1 count:        55 old count:        33 dt:      1.04 s  HEC rate:     21.14 Hz
 *** DSSSD # 2 count:       103 old count:        60 dt:      1.04 s  HEC rate:     41.32 Hz
 *** DSSSD # 3 count:        77 old count:        47 dt:      1.04 s  HEC rate:     28.83 Hz
 *** DSSSD # 4 count:        72 old count:        45 dt:      1.04 s  HEC rate:     25.95 Hz
 *** DSSSD # 5 count:        29 old count:        20 dt:      1.04 s  HEC rate:      8.65 Hz
 *** DSSSD # 6 count:         4 old count:         3 dt:      1.04 s  HEC rate:      0.96 Hz

04:01 BigRIPS and BRIKEN finish their runs
      AIDA run R1_28

04:04 BigRIPS run 1005
      BRIKEN run 5
      AIDA file R1_28

04:28 System wide checks all ok
      FEE temperatures ok

04:48 Implant hit spectra for the current degrader settings - attachment 10
      Current degrader configuration:

      Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
      Variable degraders
           1mm Al - in
           1mm Al - in
           1mm Al - out

05:00 BigRIPS and BRIKEN stop their runs
      Current AIDA file R1_42

      ASIC load check all ok

05:01 BigRips starts run 1006
      BRIKEN run 6
      AIDA file R1_42

05:03 System wide checks all ok
      FEE temps ok - attachment 11
      Good Events - attachment 12
      leakages and bias ok - attachment 13

05:37 No beam

05:40 beam back (Interlock error caused beam to be lost)

05:59 Appears to be a broken wire in DSSD2 FEE2

06:00 BigRIPS and BRIKEN stopped their runs
      AIDA currently on file R1_56

06:02 BigRIPS starts run 1007
      BRIKEN starts run 7
      AIDA currently on file R1_57

06:06 System wide checks all ok
      FEE temperatures all ok

06:27 Leakage currents still rising but within acceptable range at the moment - attachment 14

06:48 Rates from R1_67
 *** DSSSD # 1 count:    229047 old count:    159366 dt:      1.02 s  LEC rate:  68643.03 Hz
 *** DSSSD # 2 count:    275255 old count:    189329 dt:      1.02 s  LEC rate:  84646.04 Hz
 *** DSSSD # 3 count:    102832 old count:     70928 dt:      1.02 s  LEC rate:  31428.76 Hz
 *** DSSSD # 4 count:     93609 old count:     66189 dt:      1.02 s  LEC rate:  27011.55 Hz
 *** DSSSD # 5 count:     37256 old count:     26588 dt:      1.02 s  LEC rate:  10509.09 Hz
 *** DSSSD # 6 count:     69611 old count:     50443 dt:      1.02 s  LEC rate:  18882.47 Hz
 *** DSSSD # 1 count:        84 old count:        61 dt:      1.02 s  HEC rate:     22.66 Hz
 *** DSSSD # 2 count:       132 old count:       106 dt:      1.02 s  HEC rate:     25.61 Hz
 *** DSSSD # 3 count:       105 old count:        77 dt:      1.02 s  HEC rate:     27.58 Hz
 *** DSSSD # 4 count:       102 old count:        76 dt:      1.02 s  HEC rate:     25.61 Hz
 *** DSSSD # 5 count:        44 old count:        32 dt:      1.02 s  HEC rate:     11.82 Hz
 *** DSSSD # 6 count:         8 old count:         5 dt:      1.02 s  HEC rate:      2.96 Hz

07:00 BigRips and BRIKEN stop their daqs
      AIDA currently on file R1_71
      ASIC load check all ok

07:02 BigRIPS and BRIKEN start
      BigRIPS run 1008
      BRIKEN  run 8
      
      System wide checks all ok
      AIDA    run R1_71

07:30 Bias and leakage currents ok - attachment 15
      Leakage currents decreasing again

07:37 Rates from R1_79
 *** DSSSD # 1 count:    168930 old count:    115510 dt:      1.04 s  LEC rate:  51415.14 Hz
 *** DSSSD # 2 count:    203546 old count:    141908 dt:      1.04 s  LEC rate:  59324.72 Hz
 *** DSSSD # 3 count:     74562 old count:     52397 dt:      1.04 s  LEC rate:  21333.14 Hz
 *** DSSSD # 4 count:     70487 old count:     47585 dt:      1.04 s  LEC rate:  22042.48 Hz
 *** DSSSD # 5 count:     26901 old count:     18574 dt:      1.04 s  LEC rate:   8014.49 Hz
 *** DSSSD # 6 count:     51567 old count:     34847 dt:      1.04 s  LEC rate:  16092.50 Hz
 *** DSSSD # 1 count:        56 old count:        34 dt:      1.04 s  HEC rate:     21.17 Hz
 *** DSSSD # 2 count:        84 old count:        51 dt:      1.04 s  HEC rate:     31.76 Hz
 *** DSSSD # 3 count:        83 old count:        52 dt:      1.04 s  HEC rate:     29.84 Hz
 *** DSSSD # 4 count:        85 old count:        50 dt:      1.04 s  HEC rate:     33.69 Hz
 *** DSSSD # 5 count:        27 old count:        16 dt:      1.04 s  HEC rate:     10.59 Hz
 *** DSSSD # 6 count:         9 old count:         8 dt:      1.04 s  HEC rate:      0.96 Hz

08:00 BigRIPS and BRIKEN stop their daqs
      AIDA file R1_85

08:02 BigRIPS and BRIKEN start DAQs
      BigRIPS run 1009
      BRIKEN run 9
      AIDA file R1_86

08:06 System wide checks all ok
      FEE temperatures ok

08:19 Bias and leakages ok - attachment 16

Overnight and earlier today Obtained the following Merger error messages

Merge: Attempting resynchronisation to timestamp 26c3b4700056 or greater
Merge: Last sync pulse time: 26c3b46c00a0
Merge: Last timestamp seen: 26c3b4700055
Merge: Restarting at timestamp 26c3b47000a0
Merge: Timestamp (26c8062c00a0)[Q 5, Stream 5] is older than previous (26c8162bf81d) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26c8162bf81e or greater
Merge: Last sync pulse time: 26c8162800a0
Merge: Last timestamp seen: 26c8162bf81d
Merge: Restarting at timestamp 26c8162c00a0
Merge: Timestamp (26caa2109e35)[Q 5, Stream 5] is older than previous (26cab11e8415) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cab11e8416 or greater
Merge: Last sync pulse time: 26cab11c00a0
Merge: Last timestamp seen: 26cab11e8415
Merge: Restarting at timestamp 26cab12000a0
Merge: Timestamp (26cbfe8400a0)[Q 5, Stream 5] is older than previous (26cc0e83f4e5) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cc0e83f4e6 or greater
Merge: Last sync pulse time: 26cc0e8000a0
Merge: Last timestamp seen: 26cc0e83f4e5
Merge: Restarting at timestamp 26cc0e8400a0
Merge: Timestamp (26cf059000a0)[Q 5, Stream 5] is older than previous (26cf158ffb05) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cf158ffb06 or greater
Merge: Last sync pulse time: 26cf158c00a0
Merge: Last timestamp seen: 26cf158ffb05
Merge: Restarting at timestamp 26cf159000a0
Merge: Timestamp (26cf899c00a0)[Q 5, Stream 5] is older than previous (26cf999bf9ad) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26cf999bf9ae or greater
Merge: Last sync pulse time: 26cf999800a0
Merge: Last timestamp seen: 26cf999bf9ad
Merge: Restarting at timestamp 26cf999c00a0
Merge: Timestamp (26dc07cac965)[Q 5, Stream 5] is older than previous (26dc16e6255d) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26dc16e6255e or greater
Merge: Last sync pulse time: 26dc16e400a0
Merge: Last timestamp seen: 26dc16e6255d
Merge: Restarting at timestamp 26dc16e800a0
Merge: Timestamp (26e035dac9cd)[Q 5, Stream 5] is older than previous (26e043cef5cd) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26e043cef5ce or greater
Merge: Last sync pulse time: 26e043cc00a0
Merge: Last timestamp seen: 26e043cef5cd
Merge: Restarting at timestamp 26e043d000a0
Merge: Timestamp (26e20f9000a0)[Q 5, Stream 5] is older than previous (26e21f8ffbdd) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 26e21f8ffbde or greater
Merge: Last sync pulse time: 26e21f8c00a0
Merge: Last timestamp seen: 26e21f8ffbdd
Merge: Restarting at timestamp 26e21f9000a0
Merge: Timestamp (26e2817000a0)[Q 5, Stream 5] is older than previous (26e2916ffd25) [Q 5, Stream 5]

 :
 :
 :

This afternoon obtained the following error messages

MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Program (23229): check_for_data: index 6, unexpected header sequence number in pending data, found
787625409, last 27437
MERGE Program (23229): check_for_data: index 7, unexpected header sequence number in pending data, found
1814152641, last 82834
MERGE Data Link (23172): Received SEGV signal; exiting.
MERGE Server: Received signal 17.

Analysis of final two files of R33 - see attachments 1 & 2
Note idle time >> dead time (defined by PAUSE/RESUME) for nnaida5

Offline analysis R33_52 confirms ADC & disc data are synchronised - see attachments 3 & 4

aidas1> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidas1-lv_root
                       50G   42G  4.7G  91% /
tmpfs                 3.9G  1.1G  2.9G  27% /dev/shm
/dev/sda1             477M   77M  376M  17% /boot
/dev/mapper/vg_aidas1-lv_home
                      860G  580G  236G  72% /home
/dev/mapper/Data1-data10
                      7.7T  6.6T  763G  90% /data10
/dev/sdg2             7.3T  7.1T  190G  98% /ntfs
aidas1> ls /ntfs
Autorun.inf     Mar2017  NP1306    Seagate             Warranty.pdf
BackupPlus.ico  May2015  NP1512    Sept2016
Dec2016         May2016  PE16      Start_Here_Mac.app
Feb2015         Nov2016  RIBF03R1  Start_Here_Win.exe
aidas1> cd /ntfs
/ntfs
aidas1> du -h
22G	./Dec2016
41G	./Feb2015
14G	./Mar2017
123G	./May2015
7.9G	./May2016
97G	./Nov2016
1.1T	./NP1306
1.6T	./NP1512
1.7T	./PE16
101G	./RIBF03R1/BigRIPS
2.2M	./RIBF03R1/BRIKEN/1622FallonPreparation/CheckCal_161122_2208_003
25K	./RIBF03R1/BRIKEN/1622FallonPreparation/CheckCal_161124_1215
8.2G	./RIBF03R1/BRIKEN/1622FallonPreparation
500K	./RIBF03R1/BRIKEN/1626FallonParasitic/Conf
120K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id204_id184/ELOG_RIBF123_id204_id184_files
164K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id204_id184
788K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id224_id205/ELOG_RIBF123_id224_id204_files
844K	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123/elog_RIBF123_id224_id205
19M	./RIBF03R1/BRIKEN/1626FallonParasitic/Elogs_RIBF123
313M	./RIBF03R1/BRIKEN/1626FallonParasitic/FallonOnlines
187G	./RIBF03R1/BRIKEN/1626FallonParasitic
28K	./RIBF03R1/BRIKEN/BRIKENrootTools
195G	./RIBF03R1/BRIKEN
71M	./RIBF03R1/RIBF03R1_logs
2.5T	./RIBF03R1
512	./Seagate/Registration
512	./Seagate
115G	./Sept2016
0	./Start_Here_Mac.app/Contents/Frameworks
26M	./Start_Here_Mac.app/Contents/MacOS
304K	./Start_Here_Mac.app/Contents/Resources
4.0K	./Start_Here_Mac.app/Contents/_CodeSignature
26M	./Start_Here_Mac.app/Contents
26M	./Start_Here_Mac.app
7.1T	.
aidas1> 

09.00 DAQ continues OK 
      Current file May2017/R32_239

      Detector biases; leakage currents OK - see attachments 1 & 2
      FEE64 Temperatures OK - see attachment 3
      good events; statistics OK - see attachment 4
      Some error messages reported by merger yesterday but none overnight - see attachment 5

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status =6)

09.33 E17/F11 ambient temperature +24.0 deg C, RH 38.0%, d.p. +8.8 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

09.35 Analysis of R32_259 - OK see attachment 6

10.50 Offline analysis R32_277 - ADC & disc data synchronised - see attachments 7-9

11.00 DAQ stop OK 
      File May2017/R32_280

      Check ASIC control x2
      Zero all spectra

11.05 DAQ go
      File May2017/R33

      BNC PB-4
      Amplitude 90,000
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 25Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0x20

15.00 Merger crashed (SEGV) - see https://elog.ph.ed.ac.uk/AIDA/607

      DAQ stop
      Restart Merger
      DAQ stop OK following transient nnaida1 timeout

      Close merger/tapeserver browser, kill httpd for merger/tapeserver,merger and tapeserver terminal server
sessions
      Disable histogramming and data transfer #1
      DAQ reset, setup
      Restart merger and tapeserver
      Enable histogramming and data transfer #1

      *all* system wide checks OK *except* nnaida1 & nnaida21 fail ADC calibration, nnaida1 clock status=6

15.18 DAQ start
      File May2017/R34

15.25 no merger error messages
      nnaida1 & nnaida2 good events high - ~100k and ~120k respectively - see attachment 10

16.24 removed R32
      System wide checks - ASIC clock timestamp test fails on all nnaidas. Synchronised clocks. ASIC clock test
      passes.
      FEE temperatures, leakage currents - OK
      Merger reports 

      Merge: Restarting from timestamp 1 or greater
      Merge: Starting at timestamp 27d1e55c00a0
      Merge: Trying to add items to Q index 21 (Link 21) but queue already full

      Probably as a result of ADC clock synchronisation.

17:03 Used analyser on R34_23. No timewarps, negligible dead time. All other values as normal.

17.19 FEE64 temperatures, leakage currents - OK
      nnaida1 & nnaida 2 good events still high - ~100k roughly
      No further messages from merger

17.45 DAQ continues OK
      BRIKEN reports increased neutron and gamma activity. No apparent effect on AIDA rates.

18.50 DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger errors
      Used analyser on run R34_49 - no timewarps, rates as expected

19.30 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_58 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~2 s/254 s on nnaida 2 

20.00 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_63 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.3 s/255 s on nnaida 2 

20.15 BigRIPS reports beam optimisation may be over soon.
      DAQ continues OK.

20.47 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_76 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.1 s/255 s on nnaida 2 

21.49 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_90 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~2.0 s/258 s on nnaida 2 

23.05 BigRIPS optimisation ongoing. DAQ continues OK
      FEE64 temperatures, leakage currents - OK
      No merger error messages
      No panics from USB device monitor
      Used analyser on R34_109 - no timewarps, ADC data format ~ 1Mhz, other rates OK, 
                                highest dead time = ~1.9 s/256 s on nnaida 2 

ADC offset calculation using data from pulser walkthrough runs, see:
https://elog.ph.ed.ac.uk/AIDA/603

Attachment 1,2: excel and text file with tabulated ADC offset values.
Attachment 3,4: 2D histogram pulser amplitute vs channel number (with gain matching applied) for validating the
calculation.

Note: Channel with ADC offset=0 means ADC offset could not be calculated properly due to poor resolution/low
statistic

10.15 DAQ continues OK 
      Current file May2017/R31_239

      Detector biases; leakage currents OK - see attachments 1 & 2
      FEE64 Temperatures OK - see attachment 3
      good events; statistics OK - see attachment 4
      No warnings/errors reported by merger since DAQ GO - see attachment 5

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

10.30 E17/F11 ambient temperature +23.8 deg C, RH 40.9%, d.p. +9.6 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 95% - refilled)

10.35 Layouts 1, 3, 4, 7 and 8 - see attachments 6-13
      Note - as expected, zero HEC events, layouts 5 & 6 not shown

10.58 DAQ stop - transient timeout nnaida1 - followed by successful stop
      File May2017/R31_248

      Delete R31

11.12 Disable data transfer
      DAQ GO - transient timeout nnaida1 - followed by successful start 
      nnaida1 system console - see attachment 14

12:00 Moved plastic veto scintillator downstream in an attempt to reduce noise
      Rates in noisy FEEs reduced by 20-30%

14:30 DAQ continues OK
      Found that pulser induces counts in some HEC (nnaida20, 14 in particular). See attachment 15. Pulser off
removes these events completely.
      These events could be mistaken for implantation events during data acquisition.
      Increasing the fast comparator threshold HEC from 2 -> 3 suppresses the events. Increasing the threshold
to 4 does not appear to have further effects.
      Even with the increased threshold some HEC (e.g. 1.0H) still see the pulser. See attachment 16 (fast
comparator threshold HEC was 0x3).
      Checked data is synchronised in file R32_33

15:29 Set the threshold back to 2. We will perform this cut off-line in the analysis. 

13.38 DAQ continues OK 
      Current file May2017/R27_49

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (=6)

10.30 E17/F11 ambient temperature +23.8 deg C, RH 40.9%, d.p. +9.6 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

13:40 Stopped DAQ running on alpha background calibration run
      Current file May2017/R49
      tyUSB7 log (nnaida1) attached

13:43 Detector bias OFF

14.16 DAQ starts
      File May2017/R28

      Pulser walkthrough

      BNC PB-4
      Amplitude 90,000-10,000 @ 10,000 step
      x5 attenuator IN
      t_r 100ns tau_d 50us
      - polarity
      frequency 250Hz range
      delay MIN, tail pulse, INT ref

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0x20

      See attachment 6

14.25 DAQ stop OK

14.27 DAQ starts
      File May2017/R29

      Pulser walkthrough

      BNC PB-4
      Amplitude 12,000-2,000 @ 2,000 step

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0x20

      See attachment 7

14.34 DAQ stop OK

14.38 DAQ starts
      File May2017/R30

      Background, pulser OFF

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x0a
      LEC/MEC fast comparator 0x20

15.16 DAQ stop OK
      May2017/R30_8

16:17 DAQ starts okay 
      current run May2017/R31_0
      leakage current and temperature fine
      see attachment 8 and 9


  

Observe same noise structures to https://elog.ph.ed.ac.uk/AIDA/600

Extended noisy periods ~few Hz compose of fine structure with period ~350us

10.20 DAQ continues OK 
      Current file May2017/R27_22

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (=6)

10.30 E17/F11 ambient temperature +24.2 deg C, RH 39.7%, d.p. +9.7 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

      Left DAQ running on alpha background calibration run
      
      ttyUSB7 log (nnaida1) attached
      GUI got stuck while attempting to drag-and-drop ttyUSB7 to ELOG to upload. Terminal still working and
confirmed Merger working OK.
      Killed nautilus process to get GUI unstuck. Uploaded ttyUS7 via Browse -> Upload button on ELOG instead.

15.51 DAQ continues OK 
      Current file May2017/R27_28

      Detector biases; leakage currents OK - see attachment 6
      FEE64 Temperatures OK - see attachment 7
      good events; statistics OK - see attachment 8
      No warnings/errors reported by merger since DAQ GO - see attachment 9

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (=6)

15.53 E17/F11 ambient temperature +24.1 deg C, RH 40.1%, d.p. +9.8 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 75%)

      Left DAQ running on alpha background calibration run
      
      ttyUSB7 log (nnaida1) attached

19.20 DAQ continues OK 
      Current file May2017/R27_31

      Detector biases; leakage currents OK - see attachment 11 & 12
      FEE64 Temperatures OK - see attachment 13
      good events; statistics OK - see attachment 14
      No warnings/errors reported by merger since DAQ GO - see attachment 15

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)

19.30 E17/F11 ambient temperature +24.2 deg C, RH 38.5%, d.p. +9.2 deg C

Offline analysis of RIBF123R1_91 to R23_115

See https://elog.ph.ed.ac.uk/AIDA/583 & https://elog.ph.ed.ac.uk/AIDA/594

No evidence of ~1s oscillations but we do observe ~360us  oscillations in HEC-HEC, LEC-LEC and HEC-LEC times

Attachment 1 - per DSSSD HEC-HEC time 2.56us/channel - 387-246=141=361us

Attachments 2-4 - per DSSSD LEC-LEC time 2.56us/channel - 858-721=138=353us

Attachments 5-7 - per DSSSD HEC-LEC time 2.56us/channel - 699-567=132=338us

Note also 'random' spikes in s2200 DSSSD#1 HEC-LEC time. This may suggest an increase
in noise or fluctuations in noise level. 

Attachment 8 - per DSSSD m_p versus m_n - note additional feature in DSSSD#1 & #2 

Attachments 9-10 - per DSSSD HEC-LEC time 163.84us/channel - no ~1s oscillations

10.10 DAQ continues OK 
      Current file May2017/R26_17

      Detector biases; leakage currents OK - see attachment 1
      FEE64 Temperatures OK - see attachment 2
      good events; statistics OK - see attachment 3
      No warnings/errors reported by merger since DAQ GO - see attachment 4

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (=6)
      

10.17 E17/F11 ambient temperature +24.2 deg C, RH 42.8%, d.p. +10.7 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 70%)

      Lights on B2 partially off - Alvaro TD reports this may imply beam tests being carried out. He also
reports spike of gamma counts in BRIKEN clover around 10:00.

10.18 DAQ stops OK
      Current file May2017/R26_17

      Zeroed histograms. Background alpha calibration long run ongoing.

      ASIC settings 2017Mar27-16.12.39 *except*
      slow comparator 0x64
      LEC/MEC fast comparator 0x20

      BNC PB-4 settings 
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns, tau_d 50us
      polarity -
      rate 250Hz range
      delay MIN, tail pulse

10.25 DAQ stars OK
      Current file may2017/R27

      ttyUSB7 (nnaida1) attached

10.50 Measured AIDA ac mains voltage with DMM. Voltage 100.4V rms L-N
      FL11006 chiller refilled (2 litres added) 

17.05 DAQ continues OK 
      Current file May2017/R26_17

      Detector biases; leakage currents OK - see attachment 6,7
      FEE64 Temperatures OK - see attachment 8
      good events; statistics OK - see attachment 9
      No warnings/errors reported by merger since DAQ GO - see attachment 10

      *all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (=6)

17.07 E17/F11 ambient temperature +24.1 deg C, RH 39.8%, d.p. +9.5 deg C
      Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 95%)

      Left DAQ running on alpha background calibration run


      

I have tarred and zipped all 24 logs and shipped them back to the UK. Now I can browse through them.

Studying the "panics" :=

       nnaida17 has one which is an NFS mount failure.

       nnaida20 and 22 have one each which are "Starting midas:  Page fault in user mode with in_atomic() = 1 mm = c61bf200" but with different values for the mm=.

I am studying the web to understand why this may occur in a particular program. Both occur at similar places  after "Starting midas".

There are reports of "Clock not locked. status = 0xc" during Setup Electronics which is a bit unusual. These occur in nnaida4, 17 and 22.

Also   "Clock not locked. status = 0xd"  in nnaida23,17,3,19,6,5,14,7,9. Further investigation to understand all these.

The bit allocation of the Clock Status word is:- 

Bit 0 : Lock Detect bit from LMK03200 #1

Bit 1 : Lock Detect bit from LMK03200 #2

Bit 2 : Lock detect from the internal DCM for the mux clock.

Bit 3: iDelay ready signal

The LMK03200  are the two PLLs which are connected to the 50MHz input clock. The clock setup in the log directly before the error report for nnaiad17 had reported correctly setup and the ADCs had calibrated, these rely on a stable 50MHz from the LMK03200s.

I have also spotted some diagnostic information about the waveform readout that is useful.

Just a reminder..... when power cycling the equipment please leave the power off for at least 10 seconds to allow all the capacitors in the supply chain to discharge.

Also note that the console monitor window in the Pi can be used to check that all FEE64s have "finished" the power-on sequence by using the "parse" button.

Since it doesn't use the DAQ software it won't lock-up and it will give an indication of progress.

Álvaro Tolosa Delgado reports observation of c. 1s 'oscillations' in implant-decay time spectra - see below

Hello

The worst cases (oscillations in decay-curve) are:

-->R23_91_115

--> R25_142_149

Other bad cases:

--> R23_44_70

Cheers!

Check sample of data files - see attachments 1-3 - no obvious data structure issues but higher deadtime for
nnaida17 for R23_* files

Control check using file R25_12 - see attachment 4