AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 31 of 46  ELOG logo
ID Date Author Subject
  318   Thu Jun 16 08:07:37 2016 TDAIDA photographs - June 2016 - 2
Attachment 1: IMG_5842.JPG
IMG_5842.JPG
Attachment 2: IMG_5843.JPG
IMG_5843.JPG
Attachment 3: IMG_5846.JPG
IMG_5846.JPG
Attachment 4: IMG_5847.JPG
IMG_5847.JPG
  317   Thu Jun 16 08:00:10 2016 TDAIDA photographs - June 2016 - 1
Attachment 1: IMG_5838.JPG
IMG_5838.JPG
Attachment 2: IMG_5839.JPG
IMG_5839.JPG
Attachment 3: IMG_5840.JPG
IMG_5840.JPG
Attachment 4: IMG_5841.JPG
IMG_5841.JPG
  316   Thu Jun 16 03:40:43 2016 TDThursday 16 June 2016
10.40 DAQ stalled AIDA75_30 c. 01.36

      SYNC errors since baseline (start of AIDA75) - attachment 1
      SYstem-wide checks ASIC clock timestamp and clock status OK for all FEE64s

      Power-cycle & reboot

11.14 AIDA76 

      ASIC settings 2016Jun12-10.10.08
      BNC PB-4 amplitude 50,000

11.19 AIDA77

      BNC PB-4 amplitude 30,000

11.23 AIDA78

      BNC PB-4 amplitude 10,000

11.28 AIDA79

      BNC PB-4 amplitude 5,000

      All *.L histograms saved c. 11.30 Thursday 16 June 2016
      Example spectra - attachment 5

11.34 Water on floor by AIDA NIM bins/forward AIDA stand/EURICA - it would appear
      that one or more cluster LN2 cryostat inlets have thawed are or thawing - failed fill
      and/or blocked manifold?

      Julabo FL11006 water level and flow looks OK
      FEE temperatures OK - see attachment 4

      Detector bias off
      FEE64 power off
      All AIDA power off

      Source of water was determined to be wheeled air dehumdifier/cooler used to cool crates
      in 19" rack to right of AIDA (looking upstream). This had a small tray tank to collect
      water which was full to overflowing.

      2x 60Co sources (#2530 and CD395) returned to cupboard B3F (between E12 & E17 stairwells)
 
Attachment 1: 70.png
70.png
Attachment 2: 71.png
71.png
Attachment 3: 72.png
72.png
Attachment 4: 74.png
74.png
Attachment 5: 75.png
75.png
  315   Wed Jun 15 01:43:47 2016 TDWednesday 15 June 2016
09.19 DAQ stalled AIDA72_87 c. 01.25

      SYNC errors since baseline (start of AIDA72) - attachment 1

      Power-cycle & reboot

09.23 Detector biases & leakage currents -OK - attachments 2-3

09.38 System-wide checks 
      All FEE74s pass all tests *except* nnaida3 & nnaida16 fail FADC calibration
      All FEE64s but oneshow 1 SYNC error *before* DAQ GO - attachment 4

09.39 AIDA73

      ASICs checked OK
      FADC disabled
      ASIC settings 2016Jun12-09.09.37
      SyncCheck OK (EURICA-AIDA)

13.28 DAQ stalled AIDA73_73

      All FEE64s pass system-wide clock and ASIC clock tests
      SYNC errors since baseline (start of AIDA73) - attachment 5

19.06 DAQ stalled AIDA74_64

      All FEE74s pass all tests *except* FADC calibration
      SYNC errors since baseline (start of AIDA74) - attachment 6

21.47 AIDA75

      ASICs checked OK
      FADC disabled
      All FEE74s pass all tests *except* FADC calibration
      ASIC settings 2016Jun12-09.09.37
       LEC/MEC slow comparator 0x40
       LEC/MEC fast comparator 0x20
      BNC PB-4 Pulser 
       Frequency 250Hz range
      Merge ~600k data items/s (~4600kb/s)



   
Attachment 1: 60.png
60.png
Attachment 2: 61.png
61.png
Attachment 3: 62.png
62.png
Attachment 4: 63.png
63.png
Attachment 5: 65.png
65.png
Attachment 6: 66.png
66.png
  314   Tue Jun 14 01:42:56 2016 TDTuesday 14 June 2016
09.35 AIDA67 stalled AIDA67_43 (c. 01.27)
      DAQ stopped and file closed 09.32

      SYNC error counters - see attachments 1-2

      All FEE64s pass all tests *except* SYNC counter error

09.37 AIDA68

      ASICs checked

09.42 Detector bias & leakage currents, trigger rates - see attachments 3-5 

10.03 AIDA68 stopped (loss of FEE64 synchronisation)

      Power-cycle & reboot FEE64s

10.22 AIDA69 starts

      All FEE64s pass all system-wide tests *except* nnaida1 fails clock status (0x6)

10.35 SyncCheck looks OK

10.42 AIDA69_4 FEE64 synchronisation OK


13.26 AIDA69 stalled AIDA69_63(c. 13.26)

15.22 AIDA70 

      ASICs checked

15.31 AIDA70 stopped (loss of FEE64 synchronisation)

      See attachment 6

      Power-cycle & reboot FEE64s

      System-wide checks OK *except* FADC calibration nnaida6, 20, 22 & 23
      nnaida1 clock check OK
      SYNC errors baseline set

15.54 AIDA71 starts

16.04 AIDA71_1 FEE64 synchronisation OK

17.20 AIDA71_28 FEE64 synchronisation OK
      SyncCheck OK

19.55 AIDA71 DAQ stalled AIDA71_64 c. 19.02

      See attachment 7

      Power-cycle & reboot FEE64s

20.14 AIDA72 starts

      ASICs checked
      SyncCheck OK

22.13 DAQ running AIDA72_33
Attachment 1: 50.png
50.png
Attachment 2: 51.png
51.png
Attachment 3: 52.png
52.png
Attachment 4: 52.png
52.png
Attachment 5: 54.png
54.png
Attachment 6: 55.png
55.png
Attachment 7: 56.png
56.png
  313   Mon Jun 13 03:45:15 2016 TDMonday 13 June 2016
10.20 AIDA62 found stalled - no SYNCs, no data transfer
      All FEE64s pass all system-wide tests *except* nnaida3 & nnaida8 which fail ADC calibration
      Data files indicate DAQ stopped c. 06.56
      DAQ stop and file closed

10.30 Detector bias & leakage currents, trigger rates and FEE64 temperatures - see attcahments 1-4
      Julabo FL11006 set point +17 deg C (B2F/F11 ambient temperature +25.3 deg C, d.p. +11.6 deg C, RH 42.7%)

10.48 AIDA63
      Pulser walkthrough

      BNC PB4 settings

      Amplitudes 10,000-90,000 @ 10,000 step
      x5 attenuator IN
      Frequency 250Hz range 
      t_r 50ns tau_d 50us
      - polarity (+ polarity via EG&G Ortec 433A Sum & Invert Amp)
      INT ref, tail pulse, MIN delay

      ASIC settings 2016Jun12-09.09.37

      See attachment 5 showing 9x pulser peaks, noise/beta/background and low amplitude pulser-related peak

      Spectra saved to /MIDAS/SpecData c. 11.00 13 June 2016
      Note directory names are derived from internal FEE64 date/time settings which be wrong or unset

      All spectra cleared

11.06 AIDA64
     
      Per AIDA63 *except*
  
      BNC PB-4 amplitudes

      10,000-2,000 @ 2,000 step

      Spectra saved to /MIDAS/SpecData c. 11.13 13 June 2016

      Note that not all pulse peak amplitudes will be above the slow comparator threshold 

      Attachments 6-10 hit/stat spectra and example energy spectra

EURICA standalone gamma energy and efficiency measurements using 152Eu and 133Ba.
The gamma-ray sources were placed immediately upstream of DSSSD#1 inside the AIDA 
snout.

16.00 Obtained 'unable to connect' to nnaida1 through multiple power cycles/reboots and restarts
      of MERGE unable to clear problem. Finally reosrted to a reboot of aidas1 which did 'fix' the
      problem.

      AIDA65
      Brief test run  to check DAQ/merger/TapeServer - OK

16.06 AIDA66 (AIDA) 1 (EURICA)

      ~11kBq (current activity) 60Co source immediately upstream of DSSSD#1 inside the AIDA snout.

      ASIC settings 2016Jun12-09.09.37

      FADC disabled

      BNC PB-4 pulser
      Amplitude 90,000
      Frequency 25Hz range (probably ~1-few Hz)

      Attachments 11-12 Detector biases & leakage currents -OK

16.37 Attachment 13 - SyncCheck plots

16.50 FR adds ~40kBq 60Co source on top of the AIDA snout immediately above the DSSSDs
      DAQ continues

aidas1> ~/td/GREAT/analyser2 AIDA66_17
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***   SYNC100: block:     1 ptr:  5567 data: 0x82400595 module:  2 information type:  4 information field:
0x00000595 ts: 0x000000595DD400A0 OLD: ts: 0x07158F300DD4009D tag: 3 type: 0
 *** FEE64 discriminator: block: 15627 ptr: 14341 data: 0x97620008 module: 23 information type:  6 information
field: 0x00020008 ts: 0x005B300000A0 OLD: ts: 0x005B40000050 tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  161285942 (  973887.8 Hz)
        Other data format:   99834058 (  602824.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        118 (       0.7 Hz)
                               RESUME:        118 (       0.7 Hz)
                              SYNC100:      63176 (     381.5 Hz)
                           FEE64 disc:   98983953 (  597691.6 Hz)
                             MBS info:     786693 (    4750.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      26988 (     163.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      165.610 s
 FEE module #:   1 elapsed dead time       0.378 s
 FEE module #:   2 elapsed dead time       0.464 s
 FEE module #:   3 elapsed dead time       0.002 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       4.701 s
 FEE module #:   6 elapsed dead time       0.228 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.304 s
 FEE module #:  10 elapsed dead time       0.154 s
 FEE module #:  11 elapsed dead time       0.901 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.203 s
 FEE module #:  14 elapsed dead time       0.164 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.346 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:  123.023s (  260.113 blocks/s,  16.257 Mb/s)

23.00 Found DAQ stalled - zero AIDA SYNCs from all FEE64s, and data trsabfer halted
      Disk files indicate stall at c. 18.58 (AIDA66_58)

      See attachments 14-19

      All FEE64s pass all system-wide tests

      Installed new versions of check.tcl/check.tml - see attachment 20
      Collect All SYNC Error Counters for Baseline selected

23.15 Standard DAQ stop/go sequence restarts DAQ

      AIDA67
   
      No changes from AIDA66

[td@localhost 1E47-5D81]$ ~/GREAT/analyser2 AIDA67_7
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x0000028BF38C00A0 OLD: ts: 0x0DAF8F30038C0093 tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  162789406 (  931310.8 Hz)
        Other data format:   98330594 (  562544.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         81 (       0.5 Hz)
                               RESUME:         81 (       0.5 Hz)
                              SYNC100:      66680 (     381.5 Hz)
                           FEE64 disc:   97373010 (  557066.5 Hz)
                             MBS info:     890742 (    5095.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      40512 (     231.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      174.796 s
 FEE module #:   1 elapsed dead time       0.162 s
 FEE module #:   2 elapsed dead time       0.232 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.213 s
 FEE module #:   6 elapsed dead time       0.273 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.433 s
 FEE module #:  10 elapsed dead time       0.123 s
 FEE module #:  11 elapsed dead time       0.417 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.127 s
 FEE module #:  14 elapsed dead time       0.000 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.095 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.030 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:  338.012s (   94.671 blocks/s,   5.917 Mb/s)

***            ADC data: block:     1 ptr:  6003 data: 0xC63781A0 module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33184  ts: 0x************
 ***            ADC data: block:     1 ptr:  6005 data: 0xC2837F23 module: 10 fail: 0 range: 0 id:   643 
channel:  3 adc: 32547  ts: 0x************
 ***            ADC data: block:     1 ptr:  6007 data: 0xC27E7DF9 module:  9 fail: 0 range: 0 id:   638 
channel: 62 adc: 32249  ts: 0x************
 ***            ADC data: block:     1 ptr:  6009 data: 0xC1BA7D10 module:  6 fail: 0 range: 0 id:   442 
channel: 58 adc: 32016  ts: 0x************
 ***            ADC data: block:     1 ptr:  6011 data: 0xC37A7DE7 module: 13 fail: 0 range: 0 id:   890 
channel: 58 adc: 32231  ts: 0x************
 ***            ADC data: block:     1 ptr:  6013 data: 0xC62C8141 module: 24 fail: 0 range: 0 id:  1580 
channel: 44 adc: 33089  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6015 data: 0x81641120 module:  1 information type:  6 information 
field: 0x00041120 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6017 data: 0x81640806 module:  1 information type:  6 information 
field: 0x00040806 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6019 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6021 data: 0x81621000 module:  1 information type:  6 information 
field: 0x00021000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6023 data: 0x86640912 module:  6 information type:  6 information 
field: 0x00040912 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6025 data: 0x81640008 module:  1 information type:  6 information 
field: 0x00040008 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6027 data: 0x86644009 module:  6 information type:  6 information 
field: 0x00044009 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6029 data: 0x81622880 module:  1 information type:  6 information 
field: 0x00022880 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6031 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6033 data: 0x81620510 module:  1 information type:  6 information 
field: 0x00020510 ts: 0x************
 ***            ADC data: block:     1 ptr:  6035 data: 0xC17774D5 module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 29909  ts: 0x************
 ***            ADC data: block:     1 ptr:  6037 data: 0xC31480E9 module: 12 fail: 0 range: 0 id:   788 
channel: 20 adc: 33001  ts: 0x************
 ***            ADC data: block:     1 ptr:  6039 data: 0xC1BD7ED2 module:  6 fail: 0 range: 0 id:   445 
channel: 61 adc: 32466  ts: 0x************
 ***            ADC data: block:     1 ptr:  6041 data: 0xC3957E68 module: 14 fail: 0 range: 0 id:   917 
channel: 21 adc: 32360  ts: 0x************
 ***            ADC data: block:     1 ptr:  6043 data: 0xC25E7E3C module:  9 fail: 0 range: 0 id:   606 
channel: 30 adc: 32316  ts: 0x************
 ***            ADC data: block:     1 ptr:  6045 data: 0xC2777F9B module:  9 fail: 0 range: 0 id:   631 
channel: 55 adc: 32667  ts: 0x************
 ***            ADC data: block:     1 ptr:  6047 data: 0xC3627F99 module: 13 fail: 0 range: 0 id:   866 
channel: 34 adc: 32665  ts: 0x************
 ***            ADC data: block:     1 ptr:  6049 data: 0xC0C08485 module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33925  ts: 0x************
 ***            ADC data: block:     1 ptr:  6051 data: 0xC3B77E63 module: 14 fail: 0 range: 0 id:   951 
channel: 55 adc: 32355  ts: 0x************
 ***            ADC data: block:     1 ptr:  6053 data: 0xC28A7F9E module: 10 fail: 0 range: 0 id:   650 
channel: 10 adc: 32670  ts: 0x************
 ***            ADC data: block:     1 ptr:  6055 data: 0xC2B77ED9 module: 10 fail: 0 range: 0 id:   695 
channel: 55 adc: 32473  ts: 0x************
 ***            ADC data: block:     1 ptr:  6057 data: 0xC2657E47 module:  9 fail: 0 range: 0 id:   613 
channel: 37 adc: 32327  ts: 0x************
 ***            ADC data: block:     1 ptr:  6059 data: 0xC5417EB7 module: 21 fail: 0 range: 0 id:  1345 
channel:  1 adc: 32439  ts: 0x************
 ***            ADC data: block:     1 ptr:  6061 data: 0xC5777ECD module: 21 fail: 0 range: 0 id:  1399 
channel: 55 adc: 32461  ts: 0x************
 ***            ADC data: block:     1 ptr:  6063 data: 0xC3777D25 module: 13 fail: 0 range: 0 id:   887 
channel: 55 adc: 32037  ts: 0x************
 ***            ADC data: block:     1 ptr:  6065 data: 0xC2417E9B module:  9 fail: 0 range: 0 id:   577 
channel:  1 adc: 32411  ts: 0x************
 ***            ADC data: block:     1 ptr:  6067 data: 0xC63782CB module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33483  ts: 0x************
 ***            ADC data: block:     1 ptr:  6069 data: 0xC24A7F3D module:  9 fail: 0 range: 0 id:   586 
channel: 10 adc: 32573  ts: 0x************
 ***            ADC data: block:     1 ptr:  6071 data: 0xC2DC8028 module: 11 fail: 0 range: 0 id:   732 
channel: 28 adc: 32808  ts: 0x************
 ***            ADC data: block:     1 ptr:  6073 data: 0xC281801A module: 10 fail: 0 range: 0 id:   641 
channel:  1 adc: 32794  ts: 0x************
 ***            ADC data: block:     1 ptr:  6075 data: 0xC1BA7E55 module:  6 fail: 0 range: 0 id:   442 
channel: 58 adc: 32341  ts: 0x************
 ***            ADC data: block:     1 ptr:  6077 data: 0xC1817EBA module:  6 fail: 0 range: 0 id:   385 
channel:  1 adc: 32442  ts: 0x************
 ***            ADC data: block:     1 ptr:  6079 data: 0xC4B67EFE module: 18 fail: 0 range: 0 id:  1206 
channel: 54 adc: 32510  ts: 0x************
 ***            ADC data: block:     1 ptr:  6081 data: 0xC3BA7DF5 module: 14 fail: 0 range: 0 id:   954 
channel: 58 adc: 32245  ts: 0x************
 ***            ADC data: block:     1 ptr:  6083 data: 0xC0BD7E2A module:  2 fail: 0 range: 0 id:   189 
channel: 61 adc: 32298  ts: 0x************
 ***            ADC data: block:     1 ptr:  6085 data: 0xC37A7D4F module: 13 fail: 0 range: 0 id:   890 
channel: 58 adc: 32079  ts: 0x************
 ***            ADC data: block:     1 ptr:  6087 data: 0xC2837F32 module: 10 fail: 0 range: 0 id:   643 
channel:  3 adc: 32562  ts: 0x************
 ***            ADC data: block:     1 ptr:  6089 data: 0xC17779E8 module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 31208  ts: 0x************
 ***            ADC data: block:     1 ptr:  6091 data: 0xC0407D64 module:  1 fail: 0 range: 0 id:    64 
channel:  0 adc: 32100  ts: 0x************
 ***            ADC data: block:     1 ptr:  6093 data: 0xC27E7EBB module:  9 fail: 0 range: 0 id:   638 
channel: 62 adc: 32443  ts: 0x************
 ***            ADC data: block:     1 ptr:  6095 data: 0xC1BD7EA1 module:  6 fail: 0 range: 0 id:   445 
channel: 61 adc: 32417  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6097 data: 0x8B648040 module: 11 information type:  6 information 
field: 0x00048040 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6099 data: 0x8B632000 module: 11 information type:  6 information 
field: 0x00032000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6101 data: 0x8B640014 module: 11 information type:  6 information 
field: 0x00040014 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6103 data: 0x8B622000 module: 11 information type:  6 information 
field: 0x00022000 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6105 data: 0x85640100 module:  5 information type:  6 information 
field: 0x00040100 ts: 0x************
 ***            ADC data: block:     1 ptr:  6107 data: 0xC2048185 module:  8 fail: 0 range: 0 id:   516 
channel:  4 adc: 33157  ts: 0x************
 ***            ADC data: block:     1 ptr:  6109 data: 0xC3817CB5 module: 14 fail: 0 range: 0 id:   897 
channel:  1 adc: 31925  ts: 0x************
 ***            ADC data: block:     1 ptr:  6111 data: 0xC0B76DE1 module:  2 fail: 0 range: 0 id:   183 
channel: 55 adc: 28129  ts: 0x************
 ***            ADC data: block:     1 ptr:  6113 data: 0xC5347FB5 module: 20 fail: 0 range: 0 id:  1332 
channel: 52 adc: 32693  ts: 0x************
 ***            ADC data: block:     1 ptr:  6115 data: 0xC2B87EFA module: 10 fail: 0 range: 0 id:   696 
channel: 56 adc: 32506  ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6117 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6119 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x************
 *** FEE64 discriminator: block:     1 ptr:  6121 data: 0x82640003 module:  2 information type:  6 information 
field: 0x00040003 ts: 0x************
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x0000028BF38C00A0 OLD: 
ts: 0x0DAF8F30038C0093 tag: 2 type: 6
 ***   SYNC100 timestamp: block:     1 ptr:  6123 data: 0x8B4028BF module: 11 information type:  4 information 
field: 0x000028BF ts: 0x028BF38C00A0
 ***            ADC data: block:     1 ptr:  6125 data: 0xC30B80FA module: 12 fail: 0 range: 0 id:   779 
channel: 11 adc: 33018  ts: 0x028BF38C00BD
 ***            ADC data: block:     1 ptr:  6127 data: 0xC0817E77 module:  2 fail: 0 range: 0 id:   129 
channel:  1 adc: 32375  ts: 0x028BF38C00C2
 ***            ADC data: block:     1 ptr:  6129 data: 0xC277758A module:  9 fail: 0 range: 0 id:   631 
channel: 55 adc: 30090  ts: 0x028BF38C00CA
 *** FEE64 discriminator: block:     1 ptr:  6131 data: 0x8B648000 module: 11 information type:  6 information 
field: 0x00048000 ts: 0x028BF38C01D1
 *** FEE64 discriminator: block:     1 ptr:  6133 data: 0x8B640040 module: 11 information type:  6 information 
field: 0x00040040 ts: 0x028BF38C01D2
 *** FEE64 discriminator: block:     1 ptr:  6135 data: 0x8B640004 module: 11 information type:  6 information 
field: 0x00040004 ts: 0x028BF38C01D5
 *** FEE64 discriminator: block:     1 ptr:  6137 data: 0x8B640010 module: 11 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C01D9
 ***            ADC data: block:     1 ptr:  6139 data: 0xC3B77874 module: 14 fail: 0 range: 0 id:   951 
channel: 55 adc: 30836  ts: 0x028BF38C020A
 ***            ADC data: block:     1 ptr:  6141 data: 0xC0C0848E module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33934  ts: 0x028BF38C0281
 ***            ADC data: block:     1 ptr:  6143 data: 0xC63C8026 module: 24 fail: 0 range: 0 id:  1596 
channel: 60 adc: 32806  ts: 0x028BF38C02DC
 *** FEE64 discriminator: block:     1 ptr:  6145 data: 0x81641102 module:  1 information type:  6 information 
field: 0x00041102 ts: 0x028BF38C0395
 *** FEE64 discriminator: block:     1 ptr:  6147 data: 0x81640020 module:  1 information type:  6 information 
field: 0x00040020 ts: 0x028BF38C0396
 *** FEE64 discriminator: block:     1 ptr:  6149 data: 0x8B620001 module: 11 information type:  6 information 
field: 0x00020001 ts: 0x028BF38C0397
 ***            ADC data: block:     1 ptr:  6151 data: 0xC2B77CE0 module: 10 fail: 0 range: 0 id:   695 
channel: 55 adc: 31968  ts: 0x028BF38C03E1
 *** FEE64 discriminator: block:     1 ptr:  6153 data: 0x8E640200 module: 14 information type:  6 information 
field: 0x00040200 ts: 0x028BF38C03E5
 *** FEE64 discriminator: block:     1 ptr:  6155 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C046F
 *** FEE64 discriminator: block:     1 ptr:  6157 data: 0x82640009 module:  2 information type:  6 information 
field: 0x00040009 ts: 0x028BF38C0470
 *** FEE64 discriminator: block:     1 ptr:  6159 data: 0x82640002 module:  2 information type:  6 information 
field: 0x00040002 ts: 0x028BF38C0473
 ***            ADC data: block:     1 ptr:  6161 data: 0xC3777F1E module: 13 fail: 0 range: 0 id:   887 
channel: 55 adc: 32542  ts: 0x028BF38C05F3
 ***            ADC data: block:     1 ptr:  6163 data: 0xC0BA7F03 module:  2 fail: 0 range: 0 id:   186 
channel: 58 adc: 32515  ts: 0x028BF38C063A
 ***            ADC data: block:     1 ptr:  6165 data: 0xC63781EA module: 24 fail: 0 range: 0 id:  1591 
channel: 55 adc: 33258  ts: 0x028BF38C06C4
 ***            ADC data: block:     1 ptr:  6167 data: 0xC25E7EC0 module:  9 fail: 0 range: 0 id:   606 
channel: 30 adc: 32448  ts: 0x028BF38C070A
 ***            ADC data: block:     1 ptr:  6169 data: 0xC310814C module: 12 fail: 0 range: 0 id:   784 
channel: 16 adc: 33100  ts: 0x028BF38C07C5
 ***            ADC data: block:     1 ptr:  6171 data: 0xC2817FBE module: 10 fail: 0 range: 0 id:   641 
channel:  1 adc: 32702  ts: 0x028BF38C07C9
 *** FEE64 discriminator: block:     1 ptr:  6173 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C0851
 *** FEE64 discriminator: block:     1 ptr:  6175 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C0854
 ***            ADC data: block:     1 ptr:  6177 data: 0xC1817E3E module:  6 fail: 0 range: 0 id:   385 
channel:  1 adc: 32318  ts: 0x028BF38C095D
 *** FEE64 discriminator: block:     1 ptr:  6179 data: 0x86631000 module:  6 information type:  6 information 
field: 0x00031000 ts: 0x028BF38C0998
 ***            ADC data: block:     1 ptr:  6181 data: 0xC5807DFE module: 22 fail: 0 range: 0 id:  1408 
channel:  0 adc: 32254  ts: 0x028BF38C09BC
 ***     MBS information: block:     1 ptr:  6183 data: 0x8680FCC8 module:  6 information type:  8 information 
index:  0 MBS data:FCC8 ts: 0x028BF38C09F7
 ***     MBS information: block:     1 ptr:  6185 data: 0x8681680D module:  6 information type:  8 information 
index:  1 MBS data:680D ts: 0x028BF38C09F7
 ***     MBS information: block:     1 ptr:  6187 data: 0x86820007 module:  6 information type:  8 information 
index:  2 MBS data:0007 ts: 0x028BF38C09F7
 ***            ADC data: block:     1 ptr:  6189 data: 0xC0BD7F10 module:  2 fail: 0 range: 0 id:   189 
channel: 61 adc: 32528  ts: 0x028BF38C0A22
 ***            ADC data: block:     1 ptr:  6191 data: 0xC1777A9E module:  5 fail: 0 range: 0 id:   375 
channel: 55 adc: 31390  ts: 0x028BF38C0B47
 ***            ADC data: block:     1 ptr:  6193 data: 0xC3928023 module: 14 fail: 0 range: 0 id:   914 
channel: 18 adc: 32803  ts: 0x028BF38C0CFA
 ***            ADC data: block:     1 ptr:  6195 data: 0xC3427F14 module: 13 fail: 0 range: 0 id:   834 
channel:  2 adc: 32532  ts: 0x028BF38C0CFB
 *** FEE64 discriminator: block:     1 ptr:  6197 data: 0x85640100 module:  5 information type:  6 information 
field: 0x00040100 ts: 0x028BF38C0D7B
 *** FEE64 discriminator: block:     1 ptr:  6199 data: 0x85640028 module:  5 information type:  6 information 
field: 0x00040028 ts: 0x028BF38C0D7C
 *** FEE64 discriminator: block:     1 ptr:  6201 data: 0x85640020 module:  5 information type:  6 information 
field: 0x00040020 ts: 0x028BF38C0D7F
 ***            ADC data: block:     1 ptr:  6203 data: 0xC5817E5C module: 22 fail: 0 range: 0 id:  1409 
channel:  1 adc: 32348  ts: 0x028BF38C0DA4
 ***            ADC data: block:     1 ptr:  6205 data: 0xC3817F2D module: 14 fail: 0 range: 0 id:   897 
channel:  1 adc: 32557  ts: 0x028BF38C0DC2
 ***            ADC data: block:     1 ptr:  6207 data: 0xC0B76C9E module:  2 fail: 0 range: 0 id:   183 
channel: 55 adc: 27806  ts: 0x028BF38C0E0A
 *** FEE64 discriminator: block:     1 ptr:  6209 data: 0x82648000 module:  2 information type:  6 information 
field: 0x00048000 ts: 0x028BF38C0F44
 *** FEE64 discriminator: block:     1 ptr:  6211 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C0F45
 *** FEE64 discriminator: block:     1 ptr:  6213 data: 0x82630008 module:  2 information type:  6 information 
field: 0x00030008 ts: 0x028BF38C0F46
 *** FEE64 discriminator: block:     1 ptr:  6215 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C0F46
 ***            ADC data: block:     1 ptr:  6217 data: 0xC0817E94 module:  2 fail: 0 range: 0 id:   129 
channel:  1 adc: 32404  ts: 0x028BF38C0F9A
 *** FEE64 discriminator: block:     1 ptr:  6219 data: 0x82640010 module:  2 information type:  6 information 
field: 0x00040010 ts: 0x028BF38C1018
 ***            ADC data: block:     1 ptr:  6221 data: 0xC61980E8 module: 24 fail: 0 range: 0 id:  1561 
channel: 25 adc: 33000  ts: 0x028BF38C1024
 ***            ADC data: block:     1 ptr:  6223 data: 0xC6248081 module: 24 fail: 0 range: 0 id:  1572 
channel: 36 adc: 32897  ts: 0x028BF38C1024
 ***            ADC data: block:     1 ptr:  6225 data: 0xC5398017 module: 20 fail: 0 range: 0 id:  1337 
channel: 57 adc: 32791  ts: 0x028BF38C103B
 ***            ADC data: block:     1 ptr:  6227 data: 0xC4727E47 module: 17 fail: 0 range: 0 id:  1138 
channel: 50 adc: 32327  ts: 0x028BF38C1089
 ***            ADC data: block:     1 ptr:  6229 data: 0xC20080B8 module:  8 fail: 0 range: 0 id:   512 
channel:  0 adc: 32952  ts: 0x028BF38C10C1
 ***            ADC data: block:     1 ptr:  6231 data: 0xC0C08465 module:  3 fail: 0 range: 0 id:   192 
channel:  0 adc: 33893  ts: 0x028BF38C1159
 ***            ADC data: block:     1 ptr:  6233 data: 0xC2657E31 module:  9 fail: 0 range: 0 id:   613 
channel: 37 adc: 32305  ts: 0x028BF38C12C2
 *** FEE64 discriminator: block:     1 ptr:  6235 data: 0x82648010 module:  2 information type:  6 information 
field: 0x00048010 ts: 0x028BF38C12DF
 *** FEE64 discriminator: block:     1 ptr:  6237 data: 0x82640008 module:  2 information type:  6 information 
field: 0x00040008 ts: 0x028BF38C12E3
 ***            ADC data: block:     1 ptr:  6239 data: 0xC36C801D module: 13 fail: 0 range: 0 id:   876 
channel: 44 adc: 32797  ts: 0x028BF38C14CB
 *** FEE64 discriminator: block:     1 ptr:  6241 data: 0x81641000 module:  1 information type:  6 information 
field: 0x00041000 ts: 0x028BF38C153A
 *** FEE64 discriminator: block:     1 ptr:  6243 data: 0x8B632000 module: 11 information type:  6 information 
field: 0x00032000 ts: 0x028BF38C153A
 *** FEE64 discriminator: block:     1 ptr:  6245 data: 0x8B640040 module: 11 information type:  6 information 
field: 0x00040040 ts: 0x028BF38C153A
 ***            ADC data: block:     1 ptr:  6247 data: 0xC5F7816D module: 23 fail: 0 range: 0 id:  1527 
channel: 55 adc: 33133  ts: 0x028BF38C154C
 ***            ADC data: block:     1 ptr:  6249 data: 0xC3BB7E35 module: 14 fail: 0 range: 0 id:   955 
channel: 59 adc: 32309  ts: 0x028BF38C1592
 *** FEE64 discriminator: block:     1 ptr:  6251 data: 0x97630080 module: 23 information type:  6 information 
field: 0x00030080 ts: 0x028BF38C164D
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 20.png
20.png
Attachment 12: 21.png
21.png
Attachment 13: 3.png
3.png
Attachment 14: 31.png
31.png
Attachment 15: 32.png
32.png
Attachment 16: 33.png
33.png
Attachment 17: 35.png
35.png
Attachment 18: 36.png
36.png
Attachment 19: 37.png
37.png
Attachment 20: 40.png
40.png
  312   Sun Jun 12 19:19:22 2016 DKMachine Time End

3:13 operators call to tell us the machine time is over.  No more EURICA!

          AIDA62_23 ought to be the last file with legitimate physics data

          We will leave the DAQ running overnight as a b/g run.

  311   Sun Jun 12 03:05:37 2016 TD, AE, CG, MASunday 12 June 2016 part 2
BigRIPS switched from 60Ti to 64V

AIDA48 (AIDA)2034 (EURICA) 2153 (BigRIPS) 

start 08.44
stop 09.08 (FEE64 synchronisation lost -> power cycle & reboot FEE64s)

AIDA49 (AIDA) 2036 (EURICA) 2154 (BigRIPS) 

Power cycle and reboot FEE64s
FADC power down (per PJCS recommendation)
All FEE64s pass all system wide checks *except* nnaida16 which failed the ADC calibration (not significant
as waveforms are disabled and FADC powered down anyway)

start 09.30
stop 09.58

ASIC settings 2016Jun12-09.09.37


AIDA50 (AIDA) 2037(EURICA) 2155 (BigRIPS) 

start 10.02
stop 10.09 (beam lost)

ASIC settings 2016Jun12-10.10.08
  changes to nnaida15 discriminator settings to reduce deadtime (pause-resume rate)

AIDA51 (AIDA) 2038 (EURICA) 2156 (BigRIPS) 

start 10.45
stop 11.55

ASIC settings 2016Jun12-10.10.08

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA51_0
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  163625493 (  974305.9 Hz)
        Other data format:   97494507 (  580529.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1227 (       7.3 Hz)
                               RESUME:       1227 (       7.3 Hz)
                              SYNC100:      64065 (     381.5 Hz)
                           FEE64 disc:   96631002 (  575388.1 Hz)
                             MBS info:     796986 (    4745.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     213982 (    1274.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      167.941 s
 FEE module #:   1 elapsed dead time       0.142 s
 FEE module #:   2 elapsed dead time       1.125 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.307 s
 FEE module #:   6 elapsed dead time       0.579 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.082 s
 FEE module #:  10 elapsed dead time       0.112 s
 FEE module #:  11 elapsed dead time       3.204 s
 FEE module #:  12 elapsed dead time       0.005 s
 FEE module #:  13 elapsed dead time       0.212 s
 FEE module #:  14 elapsed dead time       0.426 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.046 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.064 s
 FEE module #:  21 elapsed dead time       0.319 s
 FEE module #:  22 elapsed dead time       0.248 s
 FEE module #:  23 elapsed dead time       0.010 s
 FEE module #:  24 elapsed dead time       0.190 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   60.988s (  524.691 blocks/s,  32.793 Mb/s)

See attachments 1-7 for AIDA51 rates etc

See attachment 8 for FEE64 synchronisation check (file AIDA51_0) - OK

AIDA52 (AIDA) 2039 (EURICA) 2158 (BigRIPS)  

Start 11.56
Stop 12.54 (AIDA FEE lose sync)

Settings file 2016Jun12-10.10.08
Checked correlation of FEEs in file AIDA52_0 at 12.06 and everything seems ok.

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA52_9
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  163879044 (  985540.7 Hz)
        Other data format:   97240956 (  584790.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1315 (       7.9 Hz)
                               RESUME:       1315 (       7.9 Hz)
                              SYNC100:      63433 (     381.5 Hz)
                           FEE64 disc:   96378426 (  579603.4 Hz)
                             MBS info:     796467 (    4789.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     215707 (    1297.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      166.283 s
 FEE module #:   1 elapsed dead time       0.088 s
 FEE module #:   2 elapsed dead time       0.724 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.510 s
 FEE module #:   6 elapsed dead time       1.337 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.087 s
 FEE module #:  10 elapsed dead time       0.121 s
 FEE module #:  11 elapsed dead time       2.706 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.127 s
 FEE module #:  14 elapsed dead time       0.191 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.058 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.071 s
 FEE module #:  21 elapsed dead time       0.399 s
 FEE module #:  22 elapsed dead time       0.222 s
 FEE module #:  23 elapsed dead time       0.015 s
 FEE module #:  24 elapsed dead time       0.061 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   47.215s (  677.753 blocks/s,  42.360 Mb/s)

12.20 FEE64 temperatures - see attachment 9
      Note with all FADC powered off all temperatures now green cf. attachment 8 https://elog.ph.ed.ac.uk/AIDA/305
      Julabo FL11006 set point +17 deg C (B3F ambient temperature +25.1 deg C, d.p. +11.2 deg C, RH 40.3%)

13.11 AIDA52_13 per FEE64 ADC timestamp activity OK

12.54 AIDA FEE64 lose synchronisation
      No change in system-wide checks 

AIDA53 (AIDA) 2040 (EURICA) 2159 (BigRIPS)

Power cycled and rebooted FEE64s
System-wide checks all OK except 6 failures for ADC calibration - not retried

Start 13.09
Stop 13.44 (stop to reset EURICA VME crate)

ASIC Settings file 2016Jun12-10.10.08

13.17 AIDA53_0 FEE64 sync check - OK

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA53_0
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  167386044 ( 1100597.1 Hz)
        Other data format:   93733956 (  616319.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        714 (       4.7 Hz)
                               RESUME:        714 (       4.7 Hz)
                              SYNC100:      58017 (     381.5 Hz)
                           FEE64 disc:   92936706 (  611077.6 Hz)
                             MBS info:     737805 (    4851.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     166106 (    1092.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      152.087 s
 FEE module #:   1 elapsed dead time       0.020 s
 FEE module #:   2 elapsed dead time       0.209 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.107 s
 FEE module #:   6 elapsed dead time       0.824 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.008 s
 FEE module #:   9 elapsed dead time       0.003 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.378 s
 FEE module #:  12 elapsed dead time       0.017 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.059 s
 FEE module #:  15 elapsed dead time       0.172 s
 FEE module #:  16 elapsed dead time       1.101 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.018 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.093 s
 FEE module #:  21 elapsed dead time       0.216 s
 FEE module #:  22 elapsed dead time       0.161 s
 FEE module #:  23 elapsed dead time       0.665 s
 FEE module #:  24 elapsed dead time       0.201 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   25.180s ( 1270.866 blocks/s,  79.429 Mb/s)

AIDA54 (AIDA) 2041 (EURICA) 2160 (BigRIPS) 

Start 13.46
Stop 13.53 (loss of sync AIDA54_0)

ASIC Settings file 2016Jun12-10.10.08

System-wide checks OK following loss of sync

19:25
Stable operation most of the afternoon through multiple DAQ start/stops.
FEEs remained synchronised throughout.
SYNCs lost in run AIDA57_37 leading to run stop/start.
-> in AIDA57_37 noticed relatively large chunks of data lost in nnaida6. (see attachments 10+11

19.40 Cumulative pause-resume time for AIDA57_37 (see above) and most recent
complete data file. The pause-resume times are relatively large compared to
other FEE64s but low relative to the measurement time. But we should remember
that we have observed periods of inactivity not bounded by pause-resume tags.

aidas1> ./analyser2 /TapeData/NP1512/AIDA57_37
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  164401851 (  929891.0 Hz)
        Other data format:   96718149 (  547057.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1081 (       6.1 Hz)
                               RESUME:       1081 (       6.1 Hz)
                              SYNC100:      67440 (     381.5 Hz)
                           FEE64 disc:   95842999 (  542107.9 Hz)
                             MBS info:     805548 (    4556.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     194112 (    1097.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      176.797 s
 FEE module #:   1 elapsed dead time       0.149 s
 FEE module #:   2 elapsed dead time       0.741 s
 FEE module #:   3 elapsed dead time       0.004 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.425 s
 FEE module #:   6 elapsed dead time       1.697 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.118 s
 FEE module #:  10 elapsed dead time       0.084 s
 FEE module #:  11 elapsed dead time       1.064 s
 FEE module #:  12 elapsed dead time       0.014 s
 FEE module #:  13 elapsed dead time       0.063 s
 FEE module #:  14 elapsed dead time       0.267 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.010 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.059 s
 FEE module #:  21 elapsed dead time       0.125 s
 FEE module #:  22 elapsed dead time       0.225 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.205 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   23.047s ( 1388.475 blocks/s,  86.780 Mb/s)

aidas1> ./analyser2 /TapeData/NP1512/AIDA58_6
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***            ADC data: block: 14849 ptr: 14685 data: 0xC40181C9 module: 16 fail: 0 range: 0 id:  1025 channel:  
1 adc: 33225  ts: 0x01AB000000A0 OLD: ts: 0x01AB1000003E tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  160685457 (  937303.1 Hz)
        Other data format:  100434543 (  585850.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1282 (       7.5 Hz)
                               RESUME:       1282 (       7.5 Hz)
                              SYNC100:      65397 (     381.5 Hz)
                           FEE64 disc:   99536560 (  580612.1 Hz)
                             MBS info:     830022 (    4841.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     213213 (    1243.7 Hz)

                Timewarps:        ADC:          1 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      171.434 s
 FEE module #:   1 elapsed dead time       0.091 s
 FEE module #:   2 elapsed dead time       0.870 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.107 s
 FEE module #:   6 elapsed dead time       0.518 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.058 s
 FEE module #:  10 elapsed dead time       0.095 s
 FEE module #:  11 elapsed dead time       2.397 s
 FEE module #:  12 elapsed dead time       0.055 s
 FEE module #:  13 elapsed dead time       0.157 s
 FEE module #:  14 elapsed dead time       0.238 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.013 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.174 s
 FEE module #:  21 elapsed dead time       0.298 s
 FEE module #:  22 elapsed dead time       0.280 s
 FEE module #:  23 elapsed dead time       0.004 s
 FEE module #:  24 elapsed dead time       0.071 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   18.234s ( 1754.927 blocks/s, 109.683 Mb/s)
Attachment 1: 10.png
10.png
Attachment 2: 11.png
11.png
Attachment 3: 12.png
12.png
Attachment 4: 13.png
13.png
Attachment 5: 14.png
14.png
Attachment 6: 15.png
15.png
Attachment 7: 16.png
16.png
Attachment 8: 17.png
17.png
Attachment 9: 20.png
20.png
Attachment 10: AIDA57_37-pause.png
AIDA57_37-pause.png
Attachment 11: ADIA57_37.png
ADIA57_37.png
  310   Sat Jun 11 17:24:11 2016 DK, TDSunday 12 June 2016

Runs 41 and 42 are quite short.  Run 41 was interrupted by the beam dropping (operators called), and run 42 we are just trying to get EURICA to synchronize with AIDA DAQ.

1:08 Started AIDA Run 43 for 60Ti beam (EURICA run 2027; BigRIPS run 2132; CAEN Run 3018)

          Leak currents less than 5 uA.  See attachments 1 & 2

          Scalers are shown as attachment 3

          FEEE synchronization with pulser signal is shown as attachment 4

          1:21 operator calls to note that the beam is stopped.

 

1:25 started AIDA run 44 for 60Ti beam (EURICA run 2028; BigRIPS #2133; CAEN #3019)

         1:31 stopped because CAEN DAQ goes down.

 

1:34 started AIDA run 45 for 60Ti beam (EURICA run 2029; BigRIPS #2134; CAEN #3020)

          1:37 data stops writing to disk for AIDA for unknown reason.

 

1:50 nnaida11 showed no SYNCs.  Stop / GO on Run Control resolves it.

          but 18 is off for the merger link.  was this the status quo?  try to reboot the merger but everything gets upset.

          my first power cycle alone! (daid)

          load the settings file 2016Jun10-20.30.18 which seems to be the most recent one

          it takes some time, but finally I can get things up and running it seems.

          2:40 still 18 link is down.  strange.  hopefully it's an acceptable loss, because I think it has been a long day for everyone.

          OH DUH!  nnaida19 hasn't been working for the entire experiment!!  Just I forgot on the merge links it is offset 0 and not offset 1 for counting.

          I could get some practice restoring AIDA at least?

          Anyway, there had been some problems from run 45, and now all is well.

 

2:40 AIDA run 46.  It's just a test that the system is working.  But I forgot to put the TapeServer Volume Name so who knows where it ended up (discarded to /dev/null ?)

 

2:42 AIDA run 47, started with EURICA up and ready.  (BigRIPS #2137; EUIRCA #2032; CAEN #3023)

          Basically everything looks reasonable I think

         SSD XY pattern looks basically okay.  See attachment

          Pulser shows the FEEs are synchronized nicely.  See attachment 6

          Biases and leak currents are shown in attachments 7 and 8

          Scalers are shown as attachment 9

 

Check the pause/resumes.  From the html Statistics page, nnaida20 shows some pauses, while nnaida24 seems to pause regularly.

I use Tom's program as outlined in a previous elog.  He said we shall want the elapsed dead time to be much less than the timestamp elapsed

~/td/GREAT/analyser2 /TapeData/NP1512/AIDA47_26

*** GREAT format 3.2.0 analyser - TD - May 2014
 *** FEE64 discriminator: block: 23312 ptr:  2071 data: 0x98638000 module: 24 information type:  6 information field: 0x00038000 ts: 0x0083500000A0 OLD: ts: 0x00836000009F tag: 2 type: 6
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  140888182 (  963280.2 Hz)
        Other data format:  120231818 (  822048.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       4747 (      32.5 Hz)
                               RESUME:       4747 (      32.5 Hz)
                              SYNC100:      55794 (     381.5 Hz)
                           FEE64 disc:  119436810 (  816612.9 Hz)
                             MBS info:     729720 (    4989.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     286072 (    1955.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      146.259 s
 FEE module #:   1 elapsed dead time       0.019 s
 FEE module #:   2 elapsed dead time       0.232 s
 FEE module #:   3 elapsed dead time       0.002 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.255 s
 FEE module #:   6 elapsed dead time       0.215 s
 FEE module #:   7 elapsed dead time       0.001 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.028 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.036 s
 FEE module #:  12 elapsed dead time       0.089 s
 FEE module #:  13 elapsed dead time       0.001 s
 FEE module #:  14 elapsed dead time       0.034 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.478 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.077 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.166 s
 FEE module #:  21 elapsed dead time       0.760 s
 FEE module #:  22 elapsed dead time       0.029 s
 FEE module #:  23 elapsed dead time       0.755 s
 FEE module #:  24 elapsed dead time       0.448 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

It could be around 5% total, but I do not think it is so terrible.

 

4:56 AIDA Run 47_53 begins 4:56.  We do not restart the runs.  EURICA also automatically had the new run 2033 at 4:56.  BigRIPS manually stop / start is now at 2138 (4:57).  CAEN run is still 3023.

          5:02: At start of run, the beam dropped

          5:05 PPAC F3 1A trips.

          Leak current shown as attachment 10 and 11

          Scalers shown as attachment 12

          FEE sync looks good, as well as SSD XY.  About 370 MB of ROOT tree (compared to typical check of 50 MB) Attachments 13 and 14, respectively.

 

Here are Tom's pause information

 

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA47_53
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  146454667 (  944617.0 Hz)
        Other data format:  114665333 (  739579.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       4216 (      27.2 Hz)
                               RESUME:       4216 (      27.2 Hz)
                              SYNC100:      59144 (     381.5 Hz)
                           FEE64 disc:  113857390 (  734368.1 Hz)
                             MBS info:     740367 (    4775.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     272434 (    1757.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          1 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      155.041 s
 FEE module #:   1 elapsed dead time       0.014 s
 FEE module #:   2 elapsed dead time       0.231 s
 FEE module #:   3 elapsed dead time       0.005 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.220 s
 FEE module #:   6 elapsed dead time       0.040 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.009 s
 FEE module #:   9 elapsed dead time       0.043 s
 FEE module #:  10 elapsed dead time       0.005 s
 FEE module #:  11 elapsed dead time       0.064 s
 FEE module #:  12 elapsed dead time       0.006 s
 FEE module #:  13 elapsed dead time       0.008 s
 FEE module #:  14 elapsed dead time       0.039 s
 FEE module #:  15 elapsed dead time       0.065 s
 FEE module #:  16 elapsed dead time       0.457 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.015 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.059 s
 FEE module #:  21 elapsed dead time       0.330 s
 FEE module #:  22 elapsed dead time       0.043 s
 FEE module #:  23 elapsed dead time       0.989 s
 FEE module #:  24 elapsed dead time       0.462 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   21.295s ( 1502.706 blocks/s,  93.919 Mb/s)
 

Attachment 1: bias1_AIDA43_1206-0109.png
bias1_AIDA43_1206-0109.png
Attachment 2: bias2_AIDA43_1206-01.17.png
bias2_AIDA43_1206-01.17.png
Attachment 3: Scalers_AIDA43_1206-0110.png
Scalers_AIDA43_1206-0110.png
Attachment 4: FEEE-Pulser_AIDA43_1206-01.13.png
FEEE-Pulser_AIDA43_1206-01.13.png
Attachment 5: SSD-XY_AIDA47_1206_02.53.png
SSD-XY_AIDA47_1206_02.53.png
Attachment 6: FEEE-Pulser_AIDA47_1206-02.56.png
FEEE-Pulser_AIDA47_1206-02.56.png
Attachment 7: bias1_AIDA47_1206-02.59.png
bias1_AIDA47_1206-02.59.png
Attachment 8: bias2_AIDA47_1206-03.00.png
bias2_AIDA47_1206-03.00.png
Attachment 9: Scalers_AIDA47_1206-03.00.png
Scalers_AIDA47_1206-03.00.png
Attachment 10: bias1_AIDA47_53_1206-05.01.png
bias1_AIDA47_53_1206-05.01.png
Attachment 11: bias2_AIDA47_53_1206-05.01.png
bias2_AIDA47_53_1206-05.01.png
Attachment 12: Scalers_AIDA47_53_1206-05.01.png
Scalers_AIDA47_53_1206-05.01.png
Attachment 13: FEEE-Pulser_AIDA47_53_1206-02.56.png
FEEE-Pulser_AIDA47_53_1206-02.56.png
Attachment 14: SSD-XY_AIDA47_53_1206_02.53.png
SSD-XY_AIDA47_53_1206_02.53.png
  309   Sat Jun 11 16:06:16 2016 TD[How To] Monitor per FEE64 Activity
Launch terminal and enter commands

setenv MIDASBASE /MIDAS@aidas.240215
setenv PATH $MIDASBASE/bin_Linux64:$PATH
MIDAS64-session

From the pop-up select 'MIDAS Data Analysis'

From 'MIDAS Base Frame' select 'Sort Tool' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Browse' and select 'sort2b.f' from the pop-up

From 'MIDAS Sort (64 bit) Control Frame' select 'Compile Sortfile' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Disk Info'

From 'MIDASsort Disk Info' select 'Browse' and select the latest complete file from the current 
run from the data dietcory (currently /TapeData/NP1512)

From 'MIDASsort Disk Info' select 'Add' 

From 'MIDASsort Disk Info' select 'Quit' 

From 'MIDAS Sort (64 bit) Control Frame' select 'Start'
  The terminal from which MIDAS was launched will report messages from MIDASsort
  and end of sort

From 'MIDAS Sort (64 bit) Control Frame' select 'View Spectra'

From 'Sort Spectrum Directory Browser' select spectra s3501-s3524 which display
spectra showing (ADC) timestamps per FEE (s3501 = nnaida1, s3502 = nnaida2 etc)

The 65k spectra are rolling and the scale is 1ch = 0.1s.

Examine the activity of each FEE for extended (>1s) periods of inactivity.  
  
When the sort is complete from 'MIDAS Sort (64 bit) Control Frame' select 'Pause' then 'Kill'



  
  308   Sat Jun 11 15:48:49 2016 TD[How To] Monitor Pause-Resume Times
The basic GREAT data format analyser will report per FEE cumulative
pause-resume times which should be << timestamp elapsed time (which
is time taken to write the data file analysed).

aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA40_52
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  147698907 (  924090.7 Hz)
        Other data format:  113421093 (  709628.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       6592 (      41.2 Hz)
                               RESUME:       6592 (      41.2 Hz)
                              SYNC100:      60971 (     381.5 Hz)
                           FEE64 disc:  112649552 (  704801.4 Hz)
                             MBS info:     697386 (    4363.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     312853 (    1957.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      159.832 s
 FEE module #:   1 elapsed dead time       0.030 s
 FEE module #:   2 elapsed dead time       0.291 s
 FEE module #:   3 elapsed dead time       0.029 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.132 s
 FEE module #:   6 elapsed dead time       0.078 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.041 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.099 s
 FEE module #:  12 elapsed dead time       0.038 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.008 s
 FEE module #:  15 elapsed dead time       0.074 s
 FEE module #:  16 elapsed dead time       1.036 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.026 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.247 s
 FEE module #:  21 elapsed dead time       0.421 s
 FEE module #:  22 elapsed dead time       0.024 s
 FEE module #:  23 elapsed dead time       1.628 s
 FEE module #:  24 elapsed dead time       0.493 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   26.523s ( 1206.480 blocks/s,  75.405 Mb/s)
  307   Sat Jun 11 13:57:09 2016 lauraHow to monitor and check synchronisation of FEES

Alfredo has written a program to check the synchronisation of the FEES, which need to be checked regularly

 

To do this:

 

change to directory: npg/AIDAanalysis/AIDAmonitor

Command: nice ./AIDA-sort.exe -F filename                         e.g. nice ./AIDA-sort.exe -F AIDA37_10

Be patient, it takes a while to sort the file.  It is possible to monitor the output in the directory npg/AIDAanalysis/ORDERsort/aev by the command ls -trlh /data10/NP1512.rootfiles

However, there is no need to wait for the raw date in the full 2GB to be converted to .root.  It is possible to pause it by pressing the space bar, then choosing the options (0 to stop)

Once stopped, command: nice root -l /data10/NP1512.rootfiles/aida_sort_AIDAfilename.root    e.g. nice root -l /data10/NP1512.rootfiles/aida_sort_AIDA37_10.root

Root will then open.

Command: .x checkSYNC.c

--------------------------------

Four canvases will open: (1) cxy for each DSSD, (2) cExy for each DSSD, (c) cpulser: timestamp vs NNAIDA module and (d) CADC Energy: ADC energy vs NNAIDA module

* Check in cxy that all 4 quadrants have data (excluding those corresponding to nnaida19, which is currently inactive).  If there is asymmetry (eg missing quadrants) there is probably a synchronisation problem!!

*Check in cEXY that there is a y=x correlation.  If it isn't, there is probably a synchronisation problem!!

 

If there is a synchronisation problem, it may be necessary to power cycle.  Switch all off, then sequence all on, follow AIDA elog entry 242 https://elog.ph.ed.ac.uk/AIDA/242

             

Attachment 1: eg1.png
eg1.png
Attachment 2: eg2.png
eg2.png
Attachment 3: eg3.png
eg3.png
Attachment 4: eg4.png
eg4.png
  306   Sat Jun 11 10:29:16 2016 AE, TD, LHBsynchronization between different FEEs
file AIDA10_20: stopped on June 10, 14:59
file AIDA10_40: stopped on June 10, 16:16

Time different between pulser peaks, in microseconds (ts/100), for the four groups shown in the attached plots:

AIDA10_20: 2593, 2628, 2593
AIDA10_40: 2618, 2600, 2618

Measurement consistent with a time difference of 2600 usec (there is an error of 32 usec as we're using the data
from the slow comparator)

The two runs also show the same pattern of which FEEs have the same timestamp for the pulser event


Attachment 3, for run AIDA34, corresponds to a case after we had managed to resynchronize all FEEs
Attachment 1: AIDA10_20_pulser_allign.png
AIDA10_20_pulser_allign.png
Attachment 2: AIDA10_40_pulser_allign.png
AIDA10_40_pulser_allign.png
Attachment 3: AIDA34_puls_alignm.png
AIDA34_puls_alignm.png
  305   Fri Jun 10 21:07:52 2016 AE, DK, TD, ML, MASaturday 11 June 2016
04.51 Start run AIDA18.&nbsp; Beam is back from the operators.
      Scalers are shown as attachment 1
      Leak currents seem okay (all &lt; 5 uA), shown as attachments 2 and 3.
      Discrim levels have a bit of fluctuation depending on the module -- attachment 4
      nnaida24 has some pauses (attachment 5)

09.35 AIDA20* - AIDA24* test runs with no beam, i.e. noise, betas and low rate pulser
      to investigate FEE64 synchronisation 

      AIDA20* nnaida19 data link to Merge program enabled
      AIDA21* - nnaida19 removed from DAQ configuration file (startup.tcl) and server/web browser restarted

12.35 Detector bias & leakage current, FEE temperatures - OK - see attachments 5-7
Attachment 1: Scalers-AIDA18.png
Scalers-AIDA18.png
Attachment 2: bias1_AIDA18_1106-0456.png
bias1_AIDA18_1106-0456.png
Attachment 3: bias2_AIDA18_1106-0457.png
bias2_AIDA18_1106-0457.png
Attachment 4: StatsDiscrim_AIDA18_1106-0501.png
StatsDiscrim_AIDA18_1106-0501.png
Attachment 5: StatsPause_AIDA18_1106-0506.png
StatsPause_AIDA18_1106-0506.png
Attachment 6: 1.png
1.png
Attachment 7: 2.png
2.png
Attachment 8: 3.png
3.png
  304   Fri Jun 10 14:14:56 2016 TDnnaida19
See attachments 1 and 2 and spot the difference ...

Yes - nnaida19 histogramming appears to have started working as mysteriously as it stopped.

However, although nnaida19 ReSYNCs the timestamp appears to wander (see attachment 3) and
nnaida19 still fails the ASIC timestamp and SYNC errors not counting system-wide tests.
Ideas anyone?
Attachment 1: 4.png
4.png
Attachment 2: 5.png
5.png
Attachment 3: 6.png
6.png
  303   Fri Jun 10 14:02:17 2016 TD (courtesy VFEP)[How to] Avoid Power-Cycling/Rebooting FEE64s
If you obtain the 'unable to connect' from one, or more, of the FEE64s try the
following procedure *before* power-cycling/rebooting all of the FEE64s. 

1) From the Firefox web browser (Workspace 2) close any tab related to the Merge program
   typically 'MERGE: Control', 'Merger Link Data Rates' and 'MRGE Statistics'

2) Restart the Merge program by selecting the 'MergeServer' icon on the task bar at the top
   of the display which the run the script /MIDAS/Merger/AIDA/Startup

   Restarting the Merge program in this way will ensure that all merge program components
   are removed and that the merge program starts with a timestamp of zero.

   [Attempts to just restart manually which fail to remove ALL of the link tasks will cause
    the data collection for those FEE64 links to an existing link task to fail (port already
    in use).

    It is also important that the Merge restarts with the time stamp at zero and with the link
    queues all empty  (which happens when the software is reloaded)]

3) From the Firefox web browser (Workspace 2) 'MERGE: Control' tab

   Select Reload

   The Merge state should now be 'Unknown'

   Select Configure & Setup

   Select GO

   This should immediately commence reading data from the links and sending merged data to the
   TapeServer if enabled.

   Select Toggle Merge Data Transfer State to toggle data transfer to the TapeServer 

   Select Toggle Merge Program Pause State to toggle input data to the Merge program

Based upon notes from VFEP (originally written for the IDS LyrTech system at CERN)
  302   Fri Jun 10 12:51:19 2016 TDADC offsets - pass 1 -18.15 3 June 2016
Column order:

Channel = (fee# - 1)*64 + (asic# - 1)*16 + channel
FEE
ASIC
Channel
Offset (ADC channels)
Offset error (ADC channels)
INL (%)
# peaks
Spectrum name

ADC offset to be used as follows:

 adc(i) = INT( ABS( raw_adc(i) - 32768 ) - offset(i) + 0.5 )

This is a first pass analysis with ~90% of the ADC offsets and something approaching ~95% achievable
(64 channels of nnaida19 not operational).
 
Recommend ignoring those with INL > 1%, offsets > 500 channels
Attachment 1: offsets.dat
   0  1  1  0     221.27     0.49  -0.590   9 1.0.L                           
   1  1  1  1      96.21     0.96  -0.696   7 1.1.L                           
   2  1  1  2      73.14     0.48  -0.660   9 1.2.L                           
   3  1  1  3     254.76     0.48  -0.820   9 1.3.L                           
   4  1  1  4      64.25     0.49  -0.906   9 1.4.L                           
   5  1  1  5     316.39     0.48  -0.973   9 1.5.L                           
   6  1  1  6     156.65     0.48  -0.721   9 1.6.L                           
   7  1  1  7     150.55     0.48  -0.963   9 1.7.L                           
   8  1  1  8      59.64     0.47  -0.609   9 1.8.L                           
   9  1  1  9     237.20     0.48  -0.885   9 1.9.L                           
  10  1  1 10      32.75     0.47  -0.780   9 1.10.L                          
  11  1  1 11     115.78     0.47  -0.633   9 1.11.L                          
  12  1  1 12     207.48     0.63  -0.831   8 1.12.L                          
  13  1  1 13      96.96     0.46  -0.554   9 1.13.L                          
  14  1  1 14      60.74     0.47  -0.551   9 1.14.L                          
  15  1  1 15     932.86     3.62  -2.441   3 1.15.L INL > 1%                 
  16  1  2  0    -212.45     0.46  -0.802   9 2.0.L                           
  17  1  2  1    1295.62     0.50  -0.790   8 2.1.L                           
  18  1  2  2    -468.76     3.91  -3.122   3 2.2.L  INL > 1%                 
  19  1  2  3     140.15     0.45  -0.325   9 2.3.L                           
  20  1  2  4     -43.74     0.46  -0.602   9 2.4.L                           
  21  1  2  5    -859.66     3.94  -3.295   3 2.5.L  INL > 1%                 
  22  1  2  6      -3.00     0.47  -0.449   9 2.6.L                           
  23  1  2  7     -66.06     0.45  -0.844   9 2.7.L                           
  24  1  2  8    -420.04     3.64  -4.023   3 2.8.L  INL > 1%                 
  25  1  2  9    -801.25     3.86  -3.924   3 2.9.L  INL > 1%                 
  26  1  2 10     936.38     1.26 -14.980   3 2.10.L INL > 1%                 
  27  1  2 11    -581.39     3.66  -4.018   3 2.11.L INL > 1%                 
  28  1  2 12   -1109.67     3.84  -5.496   3 2.12.L INL > 1%                 
  29  1  2 13   -8728.93     4.31  -2.102   3 2.13.L INL > 1%                 
  30  1  2 14    5373.29     1.33 -14.201   3 2.14.L INL > 1%                 
  31  1  2 15   -3621.62     3.67  -4.831   3 2.15.L INL > 1%                 
  32  1  3  0    2825.79     3.75  -6.178   3 3.0.L  INL > 1%                 
  33  1  3  1    6377.89     1.50 -12.869   3 3.1.L  INL > 1%                 
  34  1  3  2   -7968.25     2.92  -0.503   4 3.2.L                           
  35  1  3  3    1059.21     3.81  -3.203   3 3.3.L  INL > 1%                 
  36  1  3  4    1188.25     3.67  -0.424   3 3.4.L                           
  37  1  3  5     812.02     3.76  -0.411   3 3.5.L                           
  38  1  3  6     402.38     2.35  -0.884   4 3.6.L                           
  39  1  3  7      31.54     0.91  -0.573   4 3.7.L                           
  40  1  3  8    1175.06     3.21  -0.495   3 3.8.L                           
  41  1  3  9     686.01     2.24  -0.827   4 3.9.L                           
  42  1  3 10     192.03     0.46  -0.766   9 3.10.L                          
  43  1  3 11     826.79     3.71  -0.598   3 3.11.L                          
  44  1  3 12     -15.18     0.47  -0.920   9 3.12.L                          
  45  1  3 13      -8.05     0.45  -0.897   9 3.13.L                          
  46  1  3 14   -1391.29     0.55  -0.538   9 3.14.L                          
  47  1  3 15  -44213.33     3.89 -27.410   3 3.15.L INL > 1%                 
  48  1  4  0   -1231.94     1.11  -0.559   6 4.0.L                           
  49  1  4  1   -1375.25     0.73  -0.903   8 4.1.L                           
  50  1  4  2   -1343.70     0.56  -0.600   9 4.2.L                           
  51  1  4  3   -1312.69     1.16  -0.251   6 4.3.L                           
  52  1  4  4  -48308.69     4.13 -27.547   3 4.4.L  INL > 1%                 
  53  1  4  5   -1140.83     0.74  -0.618   8 4.5.L                           
  54  1  4  6   -9999.99     0.00   0.000  37 4.6.L  > 10 peaks or no data    
  56  1  4  8     313.08     0.49  -0.721   9 4.8.L                           
  57  1  4  9   -9999.99     0.00   0.000  37 4.9.L  > 10 peaks or no data    
  59  1  4 11     418.03     0.49  -0.619   9 4.11.L                          
  60  1  4 12   -9999.99     0.00   0.000  37 4.12.L > 10 peaks or no data    
  61  1  4 13     257.28     0.80  -0.701   9 4.13.L                          
  62  1  4 14   -9999.99     0.00   0.000  37 4.14.L > 10 peaks or no data    
  63  1  4 15     262.00     0.52  -0.568   9 4.15.L                          
  64  2  1  0     227.18     0.47  -0.466   9 1.0.L                           
  65  2  1  1     278.51     0.58  -0.861   9 1.1.L                           
  66  2  1  2     -38.23     0.48  -0.466   9 1.2.L                           
  67  2  1  3      62.61     0.47  -0.495   9 1.3.L                           
  68  2  1  4     209.07     0.49  -0.608   9 1.4.L                           
  69  2  1  5     164.35     0.48  -0.534   9 1.5.L                           
  70  2  1  6      33.05     0.48  -0.531   9 1.6.L                           
  71  2  1  7     468.91     0.48  -0.559   9 1.7.L                           
  72  2  1  8     232.56     0.73  -0.823   9 1.8.L                           
  73  2  1  9   -9999.99     0.00   0.000  37 1.9.L  > 10 peaks or no data    
  74  2  1 10     297.71     1.27  -0.840   7 1.10.L                          
  75  2  1 11     194.00     0.48  -0.495   9 1.11.L                          
  76  2  1 12      52.37     0.48  -0.509   9 1.12.L                          
  77  2  1 13     224.36     0.46  -0.480   9 1.13.L                          
  78  2  1 14     -19.41     0.49  -0.443   9 1.14.L                          
  79  2  1 15     -92.64     0.45  -0.347   9 1.15.L                          
  80  2  2  0     -18.96     0.45  -0.343   9 2.0.L                           
  81  2  2  1     -11.28     0.46  -0.361   9 2.1.L                           
  82  2  2  2     -55.97     0.46  -0.377   9 2.2.L                           
  83  2  2  3      56.49     0.44  -0.351   9 2.3.L                           
  84  2  2  4      66.61     0.45  -0.352   9 2.4.L                           
  85  2  2  5    -254.65     0.44  -0.388   9 2.5.L                           
  86  2  2  6     -19.33     0.44  -0.401   9 2.6.L                           
  87  2  2  7     -47.39     0.46  -0.387   9 2.7.L                           
  88  2  2  8     -35.59     0.44  -0.341   9 2.8.L                           
  89  2  2  9    -171.18     0.45  -0.405   9 2.9.L                           
  90  2  2 10      12.24     0.44  -0.368   9 2.10.L                          
  91  2  2 11       7.99     0.43  -0.375   9 2.11.L                          
  92  2  2 12    -102.26     0.45  -0.339   9 2.12.L                          
  93  2  2 13     -58.76     0.45  -0.161   9 2.13.L                          
  94  2  2 14     -16.78     0.40  -0.209   9 2.14.L                          
  95  2  2 15      54.92     0.41  -0.312   9 2.15.L                          
  96  2  3  0     433.30     0.45  -0.332   9 3.0.L                           
  97  2  3  1     142.86     0.45  -0.372   9 3.1.L                           
  98  2  3  2     114.43     0.46  -0.199   9 3.2.L                           
  99  2  3  3     312.14     0.45  -0.266   9 3.3.L                           
 100  2  3  4     185.36     0.46  -0.364   9 3.4.L                           
 101  2  3  5     137.18     0.46  -0.328   9 3.5.L                           
 102  2  3  6     258.04     0.45  -0.270   9 3.6.L                           
 103  2  3  7     284.53     0.45  -0.260   9 3.7.L                           
 104  2  3  8     115.77     0.46  -0.448   9 3.8.L                           
 105  2  3  9     285.40     0.46  -0.355   9 3.9.L                           
 106  2  3 10       2.38     0.47  -0.331   9 3.10.L                          
 107  2  3 11      28.40     0.47  -0.501   9 3.11.L                          
 108  2  3 12     228.77     0.47  -0.343   9 3.12.L                          
 109  2  3 13     318.35     0.46  -0.379   9 3.13.L                          
 110  2  3 14      76.69     0.45  -0.258   9 3.14.L                          
 111  2  3 15     222.08     0.46  -0.391   9 3.15.L                          
 112  2  4  0     263.39     0.48  -0.267   9 4.0.L                           
 113  2  4  1    -261.43     0.48  -0.353   9 4.1.L                           
 114  2  4  2    -127.19     0.47  -0.301   9 4.2.L                           
 115  2  4  3      23.73     0.48  -0.356   9 4.3.L                           
 116  2  4  4     -51.36     0.48  -0.415   9 4.4.L                           
 117  2  4  5     121.99     0.50  -0.423   9 4.5.L                           
 118  2  4  6     134.29     0.53  -0.386   9 4.6.L                           
 119  2  4  7   -9999.99     0.00   0.000  37 4.7.L  > 10 peaks or no data    
 120  2  4  8     203.54     0.49  -0.387   9 4.8.L                           
 121  2  4  9     340.38     0.51  -0.434   9 4.9.L                           
 122  2  4 10   -9999.99     0.00   0.000  37 4.10.L > 10 peaks or no data    
 123  2  4 11     129.85     0.48  -0.366   9 4.11.L                          
 124  2  4 12     210.80     0.49  -0.352   9 4.12.L                          
 125  2  4 13      42.06     0.77  -0.318   9 4.13.L                          
 126  2  4 14     171.05     0.48  -0.312   9 4.14.L                          
 127  2  4 15     -32.40     0.47  -0.307   9 4.15.L                          
 128  3  1  0    -519.64     1.60   6.862   3 1.0.L  INL > 1%                 
 129  3  1  1    -217.21     1.59   0.926   5 1.1.L                           
 130  3  1  2    -165.60     0.55   0.532   9 1.2.L                           
 131  3  1  3      11.05     0.56   0.604   9 1.3.L                           
 132  3  1  4       5.91     2.23   0.146   3 1.4.L                           
 133  3  1  5     -39.75     0.53   0.630   9 1.5.L                           
 134  3  1  6    -172.29     0.53   0.670   9 1.6.L                           
 135  3  1  7    -325.49     0.54   0.663   9 1.7.L                           
 136  3  1  8     -83.54     0.51   0.694   9 1.8.L                           
 137  3  1  9    -217.62     0.51   0.644   9 1.9.L                           
 138  3  1 10    -183.08     0.51   0.671   9 1.10.L                          
 139  3  1 11     171.84     0.52   0.665   9 1.11.L                          
 140  3  1 12    -156.49     0.50   0.557   9 1.12.L                          
 141  3  1 13    -207.29     0.52   0.583   9 1.13.L                          
 142  3  1 14     -12.61     0.50   0.522   9 1.14.L                          
 143  3  1 15    -210.33     0.49   0.528   9 1.15.L                          
 144  3  2  0    -334.03     0.48   0.572   9 2.0.L                           
 145  3  2  1    -232.98     0.46   0.567   9 2.1.L                           
 146  3  2  2    -139.49     0.48   0.570   9 2.2.L                           
 147  3  2  3    -183.13     0.46   0.605   9 2.3.L                           
 148  3  2  4    -136.91     0.46   0.521   9 2.4.L                           
 149  3  2  5     -86.13     0.47   0.590   9 2.5.L                           
 150  3  2  6     -17.87     0.47   0.587   9 2.6.L                           
 151  3  2  7    -492.32     0.46   0.734   9 2.7.L                           
 152  3  2  8    -134.59     0.46   0.715   9 2.8.L                           
 153  3  2  9    -334.83     0.48   0.564   9 2.9.L                           
 154  3  2 10     -76.65     0.46   0.788   9 2.10.L                          
 155  3  2 11      -1.23     0.47   0.623   9 2.11.L                          
 156  3  2 12    -392.22     0.47   0.563   9 2.12.L                          
 157  3  2 13    -154.27     0.48   0.617   9 2.13.L                          
 158  3  2 14    -197.34     0.50   0.195   8 2.14.L                          
 159  3  2 15     -57.01     0.42   0.571   9 2.15.L                          
 160  3  3  0    -606.06     0.87  10.505   3 3.0.L  INL > 1%                 
 161  3  3  1   -3789.71     1.68   2.064   3 3.1.L  INL > 1%                 
 162  3  3  2     -96.46     0.50   0.949   9 3.2.L                           
 163  3  3  3     -89.01     0.50   0.720   9 3.3.L                           
 164  3  3  4    -210.86     0.49   0.674   9 3.4.L                           
 165  3  3  5      20.13     0.49   0.643   9 3.5.L                           
 166  3  3  6    -824.38     1.70  12.398   3 3.6.L  INL > 1%                 
 167  3  3  7    -569.86     1.48   5.507   3 3.7.L  INL > 1%                 
 168  3  3  8    -347.62     1.28  12.958   3 3.8.L  INL > 1%                 
 169  3  3  9     -18.29     0.49   0.682   9 3.9.L                           
 170  3  3 10      -6.87     0.49   0.716   9 3.10.L                          
 171  3  3 11    -211.50     1.07  10.153   3 3.11.L INL > 1%                 
 172  3  3 12    -300.73     0.49   0.610   9 3.12.L                          
 173  3  3 13    -278.88     0.48   0.640   9 3.13.L                          
 174  3  3 14     -66.74     0.51   0.501   9 3.14.L                          
 175  3  3 15      41.47     0.53   0.493   9 3.15.L                          
 176  3  4  0   -1198.10     2.48  21.463   3 4.0.L  INL > 1%                 
 177  3  4  1    -280.37     0.51   0.505   9 4.1.L                           
 178  3  4  2     -61.89     0.53   0.416   9 4.2.L                           
 179  3  4  3     -32.21     0.48   0.560   9 4.3.L                           
 180  3  4  4    -143.35     0.55   0.558   9 4.4.L                           
 181  3  4  5    -176.09     0.50   0.683   9 4.5.L                           
 182  3  4  6    -112.67     0.49   0.708   9 4.6.L                           
 183  3  4  7   -1929.12     2.29   8.062   3 4.7.L  INL > 1%                 
 184  3  4  8      85.28     0.52   0.519   9 4.8.L                           
 185  3  4  9    -177.96     0.50   0.699   9 4.9.L                           
 186  3  4 10    -172.42     0.53   0.503   9 4.10.L                          
 187  3  4 11      -5.35     0.50   0.562   9 4.11.L                          
 188  3  4 12    -123.40     0.50   0.595   9 4.12.L                          
 189  3  4 13     -13.68     0.49   0.443   9 4.13.L                          
 190  3  4 14    -311.86     0.51   0.519   9 4.14.L                          
 191  3  4 15     -20.82     0.54   0.397   9 4.15.L                          
 192  4  1  0    -175.71     0.63   0.632   9 1.0.L                           
 193  4  1  1    -169.28     0.88   0.711   9 1.1.L                           
 194  4  1  2      36.57     0.57   0.588   9 1.2.L                           
 195  4  1  3    -142.43     0.58   0.728   9 1.3.L                           
 196  4  1  4     -60.44     0.59   0.695   9 1.4.L                           
 197  4  1  5     -46.68     0.54   0.670   9 1.5.L                           
 198  4  1  6     -91.09     0.56   0.742   9 1.6.L                           
 199  4  1  7    -244.52     0.57   0.878   9 1.7.L                           
 200  4  1  8    -136.10     0.54   0.846   9 1.8.L                           
 201  4  1  9    -139.78     0.54   0.811   9 1.9.L                           
 202  4  1 10    -189.83     0.54   0.698   9 1.10.L                          
 203  4  1 11    -103.95     0.56   0.697   9 1.11.L                          
 204  4  1 12    -145.52     0.52   0.594   9 1.12.L                          
 205  4  1 13     -77.24     0.51   0.656   9 1.13.L                          
 206  4  1 14     115.73     0.54   0.500   9 1.14.L                          
 207  4  1 15      34.41     0.53   0.546   9 1.15.L                          
 208  4  2  0      12.08     0.51   0.747   9 2.0.L                           
 209  4  2  1     -87.60     0.53   0.548   9 2.1.L                           
 210  4  2  2     165.38     0.51   0.612   9 2.2.L                           
 211  4  2  3     -92.40     0.52   0.679   9 2.3.L                           
 212  4  2  4      38.84     0.51   0.722   9 2.4.L                           
 213  4  2  5     -84.11     0.52   0.836   9 2.5.L                           
 214  4  2  6      67.80     0.50   0.758   9 2.6.L                           
 215  4  2  7    -142.00     0.56   0.634   9 2.7.L                           
 216  4  2  8     -36.78     0.53   0.647   9 2.8.L                           
 217  4  2  9     -14.18     0.50   0.709   9 2.9.L                           
 218  4  2 10    -209.22     0.51   0.787   9 2.10.L                          
 219  4  2 11     -34.54     0.48   0.758   9 2.11.L                          
 220  4  2 12      89.64     0.50   0.640   9 2.12.L                          
 221  4  2 13      86.28     0.53   0.738   9 2.13.L                          
 222  4  2 14    -248.70     0.49   0.580   9 2.14.L                          
 223  4  2 15     -89.65     0.47   0.627   9 2.15.L                          
 224  4  3  0    -541.15     1.02   7.322   3 3.0.L  INL > 1%                 
 225  4  3  1     -57.44     0.52   0.690   9 3.1.L                           
 226  4  3  2     -61.68     0.53   0.537   9 3.2.L                           
 227  4  3  3      41.12     0.53   0.504   9 3.3.L                           
 228  4  3  4      72.75     0.50   0.495   9 3.4.L                           
 229  4  3  5     134.73     0.52   0.613   9 3.5.L                           
 230  4  3  6     150.77     0.53   0.597   9 3.6.L                           
 231  4  3  7     -36.75     0.52   0.675   9 3.7.L                           
 232  4  3  8    -628.38     0.80   4.788   3 3.8.L  INL > 1%                 
 233  4  3  9    -238.74     0.52   0.659   9 3.9.L                           
 234  4  3 10       1.92     0.50   0.575   9 3.10.L                          
 235  4  3 11    -252.34     0.52   0.561   9 3.11.L                          
 236  4  3 12      52.68     0.51   0.605   9 3.12.L                          
 237  4  3 13      80.91     0.51   0.547   9 3.13.L                          
 238  4  3 14     157.18     0.51   0.553   9 3.14.L                          
 239  4  3 15    -140.31     0.51   0.441   9 3.15.L                          
 240  4  4  0    -137.72     0.52   0.634   9 4.0.L                           
 241  4  4  1     -86.83     0.51   0.641   9 4.1.L                           
 242  4  4  2      70.27     0.50   0.668   9 4.2.L                           
 243  4  4  3     -19.07     0.51   0.573   9 4.3.L                           
 244  4  4  4    -220.03     0.52   0.656   9 4.4.L                           
 245  4  4  5    -197.80     0.52   0.561   9 4.5.L                           
 246  4  4  6    -190.22     0.52   0.709   9 4.6.L                           
 247  4  4  7    -277.14     0.51   0.442   9 4.7.L                           
 248  4  4  8    -195.68     0.51   0.702   9 4.8.L                           
 249  4  4  9    -239.19     0.52   0.731   9 4.9.L                           
 250  4  4 10     -76.81     0.50   0.459   9 4.10.L                          
 251  4  4 11     -25.19     0.52   0.681   9 4.11.L                          
 252  4  4 12    -219.24     0.53   0.587   9 4.12.L                          
 253  4  4 13     -50.33     0.51   0.465   9 4.13.L                          
 254  4  4 14       1.96     0.54   0.538   9 4.14.L                          
 255  4  4 15    -331.97     0.52   0.457   9 4.15.L                          
 256  5  1  0     258.71     0.48  -0.433   9 1.0.L                           
 257  5  1  1      44.40     0.59  -0.711   9 1.1.L                           
 258  5  1  2     145.79     0.47  -0.498   9 1.2.L                           
 259  5  1  3      -2.75     0.48  -0.355   9 1.3.L                           
 260  5  1  4     342.67     0.48  -0.511   9 1.4.L                           
 261  5  1  5     278.31     0.49  -0.489   9 1.5.L                           
 262  5  1  6     117.25     0.47  -0.523   9 1.6.L                           
 263  5  1  7     253.76     0.49  -0.400   9 1.7.L                           
 264  5  1  8     275.29     0.48  -0.305   9 1.8.L                           
 265  5  1  9      87.88     0.47  -0.406   9 1.9.L                           
 266  5  1 10     129.59     0.48  -0.388   9 1.10.L                          
 267  5  1 11     120.29     0.48  -0.426   9 1.11.L                          
 268  5  1 12     180.13     0.47  -0.420   9 1.12.L                          
 269  5  1 13     148.06     0.46  -0.299   9 1.13.L                          
 270  5  1 14      74.49     0.48  -0.360   9 1.14.L                          
 271  5  1 15      21.34     0.46  -0.330   9 1.15.L                          
 273  5  2  1      78.70     0.46  -0.309   9 2.1.L                           
 274  5  2  2     349.48     0.46  -0.289   9 2.2.L                           
 275  5  2  3     -32.10     0.46  -0.364   9 2.3.L                           
 276  5  2  4     -13.87     0.45  -0.232   9 2.4.L                           
 277  5  2  5     154.01     0.47  -0.310   9 2.5.L                           
 278  5  2  6     105.69     0.45  -0.281   9 2.6.L                           
 279  5  2  7     277.13     0.45  -0.246   9 2.7.L                           
 280  5  2  8    -204.23     0.45  -0.327   9 2.8.L                           
 281  5  2  9     -42.14     0.46  -0.276   9 2.9.L                           
 282  5  2 10      44.38     0.45  -0.525   9 2.10.L                          
 284  5  2 12     -54.70     0.44  -0.248   9 2.12.L                          
 285  5  2 13     297.19     0.45  -0.552   9 2.13.L                          
 286  5  2 14     -83.88     0.43  -0.408   9 2.14.L                          
 287  5  2 15     119.39     0.39  -0.541   9 2.15.L                          
 288  5  3  0    -236.27     0.46  -0.869   9 3.0.L                           
 289  5  3  1    -329.49     0.47  -0.283   9 3.1.L                           
 290  5  3  2    -248.64     0.47  -0.348   9 3.2.L                           
 291  5  3  3    -269.61     0.49  -0.859   9 3.3.L                           
 292  5  3  4    9447.04     5.83 -33.252   3 3.4.L  INL > 1%                 
 293  5  3  5     -79.92     0.48  -0.288   9 3.5.L                           
 294  5  3  6    -143.18     0.48  -0.384   9 3.6.L                           
 295  5  3  7      53.67     0.47  -0.310   9 3.7.L                           
 296  5  3  8    -291.96     0.46  -0.384   9 3.8.L                           
 297  5  3  9    -303.34     0.48  -0.232   9 3.9.L                           
 298  5  3 10    -176.21     0.49  -0.242   9 3.10.L                          
 299  5  3 11    -132.74     0.93  -0.396   6 3.11.L                          
 300  5  3 12     -80.88     0.47  -0.155   9 3.12.L                          
 301  5  3 13    -136.35     0.46  -0.173   9 3.13.L                          
 302  5  3 14     -29.55     0.46  -0.229   9 3.14.L                          
 303  5  3 15    -653.05     0.95  -0.302   6 3.15.L                          
... 1217 more lines ...
  301   Fri Jun 10 08:04:11 2016 YSLeakage Currents Spread Sheet

Leakage Currents Spread Sheet and Chart

Attachment 1: AIDA_leakage_current.ods
Attachment 2: 44.png
44.png
  300   Fri Jun 10 02:52:02 2016 TD, ML & YSFriday 10 June 2016
10.50 SyncCheck ... EURICA server refusing connection

[aida@d05 euricatssender]$ ./euricatssender -t
euricatssender
Setting Transfer Block Size 65536
TCP transfer library version 4.0T
0: TCP socket send buffer was 16384 - now 425984
0: TCP socket receive buffer was 87380 - now 425984
0: TCP socket created OK - now connecting to 10.32.0.12 port 10305
0: Connected to 10.32.0.12 port 10305
E-f_evcli: Error connecting node:e01, port:6003. Msg:
-I- f_stc connection refused.
Can't connect Event Server = e01

11.08 Run AIDA9
      Stat/rate hit patterns - see attachments 1-3 

11.12 B3F ambient temp +25.0 deg C, d.p. +12.2 deg C, RH 44.2%

12.10 Attachments 4-9 - Right HEC 1*H spectra for DSSSD #1 - #6

12.17 Attachments 10-13 - nnaida24 (DSSSD #2) ASIC # 1-4 *H spectra

12.49 System wide checks OK 
      Some failures of FADC calibration - OK - readout is currently disabled
      nnaida19 fails SYNC errors not counting & ASIC clock timestamp

13.40 Run AIDA10

14.46 AIDA10_14 deadtime check

 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***            ADC data: block: 30030 ptr: 12945 data: 0xC4A87DD1 module: 18 fail: 0 range: 0 id:  1192 
channel: 40 adc: 32209  ts: 0x0F0AC00000A0 OLD: ts: 0x0F0AD0000039 tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  204745037 (  910555.1 Hz)
        Other data format:   56374963 (  250714.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        332 (       1.5 Hz)
                               RESUME:        332 (       1.5 Hz)
                              SYNC100:      85777 (     381.5 Hz)
                           FEE64 disc:   56286977 (  250323.0 Hz)
                             MBS info:       1545 (       6.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     177071 (     787.5 Hz)

                Timewarps:        ADC:          1 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      224.857 s
 FEE module #:   1 elapsed dead time       0.010 s
 FEE module #:   2 elapsed dead time       0.209 s
 FEE module #:   3 elapsed dead time       0.001 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.120 s
 FEE module #:   6 elapsed dead time       0.362 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.019 s
 FEE module #:  10 elapsed dead time       0.011 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.005 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.000 s
 FEE module #:  15 elapsed dead time       0.059 s
 FEE module #:  16 elapsed dead time       0.304 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.010 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.029 s
 FEE module #:  21 elapsed dead time       0.035 s
 FEE module #:  22 elapsed dead time       0.006 s
 FEE module #:  23 elapsed dead time       0.052 s
 FEE module #:  24 elapsed dead time       0.044 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   68.563s (  466.727 blocks/s,  29.170 Mb/s)

21.08 Detector bias and leakage current screenshots - attachments 14-15

21.36 FEE64 temperatures - attachment 16
      Julabo FL11006 set point +17 deg C (B3F temperature +25.0 deg C, d.p. +11.5 deg C, RH 42.4%)
         
Attachment 1: 60.png
60.png
Attachment 2: 61.png
61.png
Attachment 3: 62.png
62.png
Attachment 4: 61.png
61.png
Attachment 5: 62.png
62.png
Attachment 6: 63.png
63.png
Attachment 7: 64.png
64.png
Attachment 8: 66.png
66.png
Attachment 9: 67.png
67.png
Attachment 10: 81.png
81.png
Attachment 11: 82.png
82.png
Attachment 12: 83.png
83.png
Attachment 13: 84.png
84.png
Attachment 14: 1.png
1.png
Attachment 15: 2.png
2.png
Attachment 16: 3.png
3.png
  299   Fri Jun 10 02:25:45 2016 TD, ML & YSDAQ State of Play - Friday 10 June 2016
10.25 Screenshots showing DAQ state of play during NP1512
Attachment 1: 30.png
30.png
Attachment 2: 41.png
41.png
Attachment 3: 42.png
42.png
Attachment 4: 44.png
44.png
Attachment 5: 91.png
91.png
Attachment 6: 92.png
92.png
ELOG V3.1.4-unknown