AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  AIDA, Page 14 of 46  ELOG logo
ID Date Authordown Subject
  353   Thu Sep 1 03:45:34 2016 TDThursday 1 September
13.33 R11 starts

      ASIC settings 2016Aug20-15.34.53
      Discriminator masks -> 0x0
      Slow comparator 0x10
      LEC/MEC fast comparator 0xff
      From Correlation Status & Control tab change Trigger delay 0x180

      BNC PB-4 Pulser
      Amplitude 90,000
      x5 attenuator IN
      t_r 50ns tau_d 50us 
      - polarity
      INT ref, delay MIN, tail pulse
      Frequency c. 5Hz

      B2F/F11 temp +24.8 deg C, d.p. +11.9 deg C, RH 44.8%

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK - see attachments 4-9

      AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5 & nnaida20 statistics - see 
attachments 10-20
      Good event rate varies from 5-320k / FEE64

      Merger/tapeserver - see attachments 21-22
      Merged data rate ~2-3M data items/s, c. 16M to disk 

Analysis of R11_93 - FEE64 data synchronised - see attachment 23

aidas1> ./analyser2 /TapeData/Aug2016/R11_93
 *** GREAT format 3.2.0 analyser - TD - May 2014
 ***            ADC data: block:  3402 ptr: 15985 data: 0xC4C37FDB module: 19 fail: 0 range: 0 id:  1219 channel:  3 adc: 32731  ts:
0x00000120C00000A0 OLD: ts: 0x00000120D000008D tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  170966247 ( 1387907.5 Hz)
        Other data format:   90121753 (  731610.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        106 (       0.9 Hz)
                               RESUME:        106 (       0.9 Hz)
                              SYNC100:    1124303 (    9127.1 Hz)
                           FEE64 disc:   88996299 (  722473.8 Hz)
                             MBS info:        939 (       7.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        571 (       4.6 Hz)

                Timewarps:        ADC:          1 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      123.183 s
 FEE module #:   1 elapsed dead time       0.370 s
 FEE module #:   2 elapsed dead time       0.317 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       1.058 s
 FEE module #:   6 elapsed dead time       2.581 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.130 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       1.078 s
 FEE module #:  14 elapsed dead time       0.172 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.223 s
 FEE module #:  18 elapsed dead time       0.017 s
 FEE module #:  19 elapsed dead time       0.089 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       2.291 s
 FEE module #:  22 elapsed dead time       0.450 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   21.035s ( 1521.263 blocks/s,  95.079 Mb/s)

Verbose data dump around time warp (file R11_93, block 3402, ptr 15985)

 ***            ADC data: block:  3402 ptr: 15901 data: 0xC5777559 module: 21 fail: 0 range: 0 id:  1399 channel: 55 adc: 30041  ts:
0x00000120CFFFF895
 *** FEE64 discriminator: block:  3402 ptr: 15903 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFF895
 ***            ADC data: block:  3402 ptr: 15905 data: 0xC17A793F module:  5 fail: 0 range: 0 id:   378 channel: 58 adc: 31039  ts:
0x00000120CFFFF895
 *** FEE64 discriminator: block:  3402 ptr: 15907 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFF898
 *** FEE64 discriminator: block:  3402 ptr: 15909 data: 0x82640080 module:  2 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFF89A
 *** FEE64 discriminator: block:  3402 ptr: 15911 data: 0x81640080 module:  1 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFF89B
 ***            ADC data: block:  3402 ptr: 15913 data: 0xC5018140 module: 20 fail: 0 range: 0 id:  1281 channel:  1 adc: 33088  ts:
0x00000120CFFFF8D5
 ***            ADC data: block:  3402 ptr: 15915 data: 0xC3617FA1 module: 13 fail: 0 range: 0 id:   865 channel: 33 adc: 32673  ts:
0x00000120CFFFF92D
 ***            ADC data: block:  3402 ptr: 15917 data: 0xC5417DDE module: 21 fail: 0 range: 0 id:  1345 channel:  1 adc: 32222  ts:
0x00000120CFFFF95D
 ***            ADC data: block:  3402 ptr: 15919 data: 0xC57A7D7B module: 21 fail: 0 range: 0 id:  1402 channel: 58 adc: 32123  ts:
0x00000120CFFFF95D
 ***            ADC data: block:  3402 ptr: 15921 data: 0xC3687F55 module: 13 fail: 0 range: 0 id:   872 channel: 40 adc: 32597  ts:
0x00000120CFFFF9F5
 *** FEE64 discriminator: block:  3402 ptr: 15923 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFB5B
 *** FEE64 discriminator: block:  3402 ptr: 15925 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFB5C
 *** FEE64 discriminator: block:  3402 ptr: 15927 data: 0x82640080 module:  2 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFB5F
 ***            ADC data: block:  3402 ptr: 15929 data: 0xC35F7DAD module: 13 fail: 0 range: 0 id:   863 channel: 31 adc: 32173  ts:
0x00000120CFFFFB85
 ***            ADC data: block:  3402 ptr: 15931 data: 0xC3307F91 module: 12 fail: 0 range: 0 id:   816 channel: 48 adc: 32657  ts:
0x00000120CFFFFBB5
 ***            ADC data: block:  3402 ptr: 15933 data: 0xC5A67F6D module: 22 fail: 0 range: 0 id:  1446 channel: 38 adc: 32621  ts:
0x00000120CFFFFBDD
 ***            ADC data: block:  3402 ptr: 15935 data: 0xC0C0845E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 33886  ts:
0x00000120CFFFFC35
 ***            ADC data: block:  3402 ptr: 15937 data: 0xC3657ED9 module: 13 fail: 0 range: 0 id:   869 channel: 37 adc: 32473  ts:
0x00000120CFFFFC4D
 *** FEE64 discriminator: block:  3402 ptr: 15939 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFC77
 *** FEE64 discriminator: block:  3402 ptr: 15941 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFC7A
 *** FEE64 discriminator: block:  3402 ptr: 15943 data: 0x81640080 module:  1 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFC7C
 *** FEE64 discriminator: block:  3402 ptr: 15945 data: 0x82640080 module:  2 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFC7C
 ***            ADC data: block:  3402 ptr: 15947 data: 0xC4DC8041 module: 19 fail: 0 range: 0 id:  1244 channel: 28 adc: 32833  ts:
0x00000120CFFFFCB8
 *** FEE64 discriminator: block:  3402 ptr: 15949 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFCC6
 ***            ADC data: block:  3402 ptr: 15951 data: 0xC38C7EF8 module: 14 fail: 0 range: 0 id:   908 channel: 12 adc: 32504  ts:
0x00000120CFFFFD3D
 ***            ADC data: block:  3402 ptr: 15953 data: 0xC4817ED8 module: 18 fail: 0 range: 0 id:  1153 channel:  1 adc: 32472  ts:
0x00000120CFFFFD85
 ***            ADC data: block:  3402 ptr: 15955 data: 0xC3577EE4 module: 13 fail: 0 range: 0 id:   855 channel: 23 adc: 32484  ts:
0x00000120CFFFFDDD
 ***            ADC data: block:  3402 ptr: 15957 data: 0xC1B67F41 module:  6 fail: 0 range: 0 id:   438 channel: 54 adc: 32577  ts:
0x00000120CFFFFE35
 *** FEE64 discriminator: block:  3402 ptr: 15959 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFF44
 ***            ADC data: block:  3402 ptr: 15961 data: 0xC5517FCD module: 21 fail: 0 range: 0 id:  1361 channel: 17 adc: 32717  ts:
0x00000120CFFFFF9D
 ***            ADC data: block:  3402 ptr: 15963 data: 0xC5957FEA module: 22 fail: 0 range: 0 id:  1429 channel: 21 adc: 32746  ts:
0x00000120CFFFFFC5
 ***            ADC data: block:  3402 ptr: 15965 data: 0xC5B37E90 module: 22 fail: 0 range: 0 id:  1459 channel: 51 adc: 32400  ts:
0x00000120CFFFFFC5
 *** FEE64 discriminator: block:  3402 ptr: 15967 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120CFFFFFD9
 ***            ADC data: block:  3402 ptr: 15969 data: 0xC4B77FCE module: 18 fail: 0 range: 0 id:  1207 channel: 55 adc: 32718  ts:
0x00000120CFFFFFDD
 ***            ADC data: block:  3402 ptr: 15971 data: 0xC5047FBE module: 20 fail: 0 range: 0 id:  1284 channel:  4 adc: 32702  ts:
0x00000120CFFFFFDD
 ***            ADC data: block:  3402 ptr: 15973 data: 0xC3627FFD module: 13 fail: 0 range: 0 id:   866 channel: 34 adc: 32765  ts:
0x00000120D0000035
 *** FEE64 discriminator: block:  3402 ptr: 15975 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D0000059
 *** FEE64 discriminator: block:  3402 ptr: 15977 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120D000005D
 *** FEE64 discriminator: block:  3402 ptr: 15979 data: 0x82640080 module:  2 information type:  6 information field: 0x00040080 ts:
0x00000120D000005F
 ***            ADC data: block:  3402 ptr: 15981 data: 0xC5437FC7 module: 21 fail: 0 range: 0 id:  1347 channel:  3 adc: 32711  ts:
0x00000120D0000065
 ***            ADC data: block:  3402 ptr: 15983 data: 0xC583802E module: 22 fail: 0 range: 0 id:  1411 channel:  3 adc: 32814  ts:
0x00000120D000008D
 ***            ADC data: block:  3402 ptr: 15985 data: 0xC4C37FDB module: 19 fail: 0 range: 0 id:  1219 channel:  3 adc: 32731  ts:
0x00000120C00000A0 OLD: ts: 0x00000120D000008D tag: 3 type: 0
 ***            ADC data: block:  3402 ptr: 15985 data: 0xC4C37FDB module: 19 fail: 0 range: 0 id:  1219 channel:  3 adc: 32731  ts:
0x00000120C00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15987 data: 0x8140120D module:  1 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15989 data: 0x8240120D module:  2 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15991 data: 0x8340120D module:  3 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15993 data: 0x8440120D module:  4 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15995 data: 0x8540120D module:  5 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15997 data: 0x8640120D module:  6 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 15999 data: 0x8740120D module:  7 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16001 data: 0x8840120D module:  8 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16003 data: 0x8940120D module:  9 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16005 data: 0x8A40120D module: 10 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16007 data: 0x8B40120D module: 11 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16009 data: 0x8C40120D module: 12 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16011 data: 0x8D40120D module: 13 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16013 data: 0x8E40120D module: 14 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16015 data: 0x8F40120D module: 15 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16017 data: 0x9040120D module: 16 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16019 data: 0x9140120D module: 17 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16021 data: 0x9240120D module: 18 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16023 data: 0x9340120D module: 19 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16025 data: 0x9440120D module: 20 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16027 data: 0x9540120D module: 21 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16029 data: 0x9640120D module: 22 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16031 data: 0x9740120D module: 23 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***   SYNC100 timestamp: block:  3402 ptr: 16033 data: 0x9840120D module: 24 information type:  4 information field: 0x0000120D ts:
0x00000120D00000A0
 ***            ADC data: block:  3402 ptr: 16035 data: 0xC4BA7DF1 module: 18 fail: 0 range: 0 id:  1210 channel: 58 adc: 32241  ts:
0x00000120D00000A5
 *** FEE64 discriminator: block:  3402 ptr: 16037 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D00000A9
 *** FEE64 discriminator: block:  3402 ptr: 16039 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120D00000AC
 ***            ADC data: block:  3402 ptr: 16041 data: 0xC5F7812E module: 23 fail: 0 range: 0 id:  1527 channel: 55 adc: 33070  ts:
0x00000120D000010D
 ***            ADC data: block:  3402 ptr: 16043 data: 0xC5467E32 module: 21 fail: 0 range: 0 id:  1350 channel:  6 adc: 32306  ts:
0x00000120D000012D
 ***            ADC data: block:  3402 ptr: 16045 data: 0xC4C4801A module: 19 fail: 0 range: 0 id:  1220 channel:  4 adc: 32794  ts:
0x00000120D0000168
 ***            ADC data: block:  3402 ptr: 16047 data: 0xC54F7E07 module: 21 fail: 0 range: 0 id:  1359 channel: 15 adc: 32263  ts:
0x00000120D00001F5
 ***            ADC data: block:  3402 ptr: 16049 data: 0xC1907F13 module:  6 fail: 0 range: 0 id:   400 channel: 16 adc: 32531  ts:
0x00000120D000021D
 ***            ADC data: block:  3402 ptr: 16051 data: 0xC1B76F59 module:  6 fail: 0 range: 0 id:   439 channel: 55 adc: 28505  ts:
0x00000120D000021D
 *** FEE64 discriminator: block:  3402 ptr: 16053 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D0000268
 ***            ADC data: block:  3402 ptr: 16055 data: 0xC519804A module: 20 fail: 0 range: 0 id:  1305 channel: 25 adc: 32842  ts:
0x00000120D00002FD
 *** FEE64 discriminator: block:  3402 ptr: 16057 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D0000326
 ***            ADC data: block:  3402 ptr: 16059 data: 0xC3A57F86 module: 14 fail: 0 range: 0 id:   933 channel: 37 adc: 32646  ts:
0x00000120D000037D
 ***            ADC data: block:  3402 ptr: 16061 data: 0xC5707DD1 module: 21 fail: 0 range: 0 id:  1392 channel: 48 adc: 32209  ts:
0x00000120D0000385
 ***            ADC data: block:  3402 ptr: 16063 data: 0xC5B57F01 module: 22 fail: 0 range: 0 id:  1461 channel: 53 adc: 32513  ts:
0x00000120D00003AD
 ***            ADC data: block:  3402 ptr: 16065 data: 0xC2457E6F module:  9 fail: 0 range: 0 id:   581 channel:  5 adc: 32367  ts:
0x00000120D00003BD
 *** FEE64 discriminator: block:  3402 ptr: 16067 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D00003BD
 ***            ADC data: block:  3402 ptr: 16069 data: 0xC4C181BD module: 19 fail: 0 range: 0 id:  1217 channel:  1 adc: 33213  ts:
0x00000120D00003C0
 ***            ADC data: block:  3402 ptr: 16071 data: 0xC4ED80E8 module: 19 fail: 0 range: 0 id:  1261 channel: 45 adc: 33000  ts:
0x00000120D00003C0
 ***            ADC data: block:  3402 ptr: 16073 data: 0xC4F6812B module: 19 fail: 0 range: 0 id:  1270 channel: 54 adc: 33067  ts:
0x00000120D00003C0
 ***            ADC data: block:  3402 ptr: 16075 data: 0xC3457F08 module: 13 fail: 0 range: 0 id:   837 channel:  5 adc: 32520  ts:
0x00000120D000041D
 ***            ADC data: block:  3402 ptr: 16077 data: 0xC37C7FB1 module: 13 fail: 0 range: 0 id:   892 channel: 60 adc: 32689  ts:
0x00000120D000041D
 *** FEE64 discriminator: block:  3402 ptr: 16079 data: 0x85640080 module:  5 information type:  6 information field: 0x00040080 ts:
0x00000120D000043B
 *** FEE64 discriminator: block:  3402 ptr: 16081 data: 0x86640080 module:  6 information type:  6 information field: 0x00040080 ts:
0x00000120D000043B
 *** FEE64 discriminator: block:  3402 ptr: 16083 data: 0x81640080 module:  1 information type:  6 information field: 0x00040080 ts:
0x00000120D000043F
 *** FEE64 discriminator: block:  3402 ptr: 16085 data: 0x82640080 module:  2 information type:  6 information field: 0x00040080 ts:
0x00000120D0000440
 ***            ADC data: block:  3402 ptr: 16087 data: 0xC38A7F89 module: 14 fail: 0 range: 0 id:   906 channel: 10 adc: 32649  ts:
0x00000120D0000445
 ***            ADC data: block:  3402 ptr: 16089 data: 0xC3A68005 module: 14 fail: 0 range: 0 id:   934 channel: 38 adc: 32773  ts:
0x00000120D0000445
 ***            ADC data: block:  3402 ptr: 16091 data: 0xC3B07E25 module: 14 fail: 0 range: 0 id:   944 channel: 48 adc: 32293  ts:
0x00000120D0000445
 ***            ADC data: block:  3402 ptr: 16093 data: 0xC5737ED1 module: 21 fail: 0 range: 0 id:  1395 channel: 51 adc: 32465  ts:
0x00000120D000044D
 ***            ADC data: block:  3402 ptr: 16095 data: 0xC5A17F32 module: 22 fail: 0 range: 0 id:  1441 channel: 33 adc: 32562  ts:
0x00000120D0000475
 ***            ADC data: block:  3402 ptr: 16097 data: 0xC4FB80F3 module: 19 fail: 0 range: 0 id:  1275 channel: 59 adc: 33011  ts:
0x00000120D0000488
 ***            ADC data: block:  3402 ptr: 16099 data: 0xC37F7E9C module: 13 fail: 0 range: 0 id:   895 channel: 63 adc: 32412  ts:
0x00000120D00004E5
 ***            ADC data: block:  3402 ptr: 16101 data: 0xC1837FED module:  6 fail: 0 range: 0 id:   387 channel:  3 adc: 32749  ts:
0x00000120D000053D

Note - system-wide checks still OK despite time warp observed above
       no errors reported by merger since start of R11


Free memory before and after saving all L and H spectra to disk - see attachments 24-25
Some merger errors observed during save of all L and H spectra to disk - see attachment 26

FEE64 temperatures - Julabo FL11006 set point + 20 deg C - see attachment 27
                                    set point + 16 deg C - see attachment 28
Temperatures decrease typically ~ 2-4 deg C which implies that the coolant loop is working as expected.
However, +16 deg C is marginal due to relative humidity variations ~40-55% corresponding to Dew Point
temperatures of ~12-14 deg C. Returning Julabo FL11006 set point to +20 deg C.
  354   Fri Sep 2 06:22:26 2016 TDFriday 2 September
14.20 R11 continues

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)
      Free memory - see attachment 4

      Merger errors messages for links 4 & 5 - see attachment 5

     AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5 &
nnaida20 statistics - see 
attachments 6-16

Analysis of R11_645 - FEE64 data synchronised - see attachment 17

aidas1> ./analyser2 /TapeData/Aug2016/R11_645
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  156678476 ( 1098367.0 Hz)
        Other data format:  104409524 (  731944.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         92 (       0.6 Hz)
                               RESUME:         92 (       0.6 Hz)
                              SYNC100:    1303563 (    9138.4 Hz)
                           FEE64 disc:  103104691 (  722797.4 Hz)
                             MBS info:       1086 (       7.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        650 (       4.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      142.647 s
 FEE module #:   1 elapsed dead time       0.186 s
 FEE module #:   2 elapsed dead time       0.427 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       1.001 s
 FEE module #:   6 elapsed dead time       3.508 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.053 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.100 s
 FEE module #:  14 elapsed dead time       0.164 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.264 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.028 s
 FEE module #:  22 elapsed dead time       0.171 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   15.730s ( 2034.269 blocks/s, 127.142 Mb/s)

14.54 R11 stopped (two DAQ STOP commands required)

15.00 New firmware version 8.24 installed (/MIDAS/Aida/FEE_GF_Feb16_24.bin) 
      - guarantee SYNC is the last data item with timestamp ending 0x000000A0 to be output (merger assumes this)
      - ASIC clock runs when output disabled (ASIC requires periodic reset)

18.40 R13 starts

      ASIC settings 2016Sep02-18.17.01
      Discriminator masks -> 0x0
      Correlation scaler trigger delay 0x180 -> 0x5
  355   Sat Sep 3 05:54:14 2016 TDSaturday 3 September
14.20 R13 continues

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

      Merger errors messages for Q 19 - see attachment 4

     AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5 &
nnaida20 statistics - see attachments 5-15

     Analysis of R13_317 - FEE64 data synchronised - see attachment 16

aidas1> ./analyser2 /TapeData/Aug2016/R13_317
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  107206849 (  490913.0 Hz)
        Other data format:  153881151 (  704640.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        134 (       0.6 Hz)
                               RESUME:        134 (       0.6 Hz)
                              SYNC100:    1997059 (    9144.8 Hz)
                           FEE64 disc:  151882150 (  695486.5 Hz)
                             MBS info:       1674 (       7.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       1095 (       5.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      218.383 s
 FEE module #:   1 elapsed dead time       0.437 s
 FEE module #:   2 elapsed dead time       0.298 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       2.671 s
 FEE module #:   6 elapsed dead time       1.429 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.000 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.129 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.010 s
 FEE module #:  18 elapsed dead time       0.000 s
 FEE module #:  19 elapsed dead time       0.392 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.021 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   19.699s ( 1624.430 blocks/s, 101.527 Mb/s)

18.30 HDMI cabling to NIM MACB modules mechanically secured by 1U 'cable organiser' installed above MACB NIM bin
and by AIDA stand frame
  
      ALL system-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

18.55 R16 starts

      ASIC settings 2016Sep02-18.17.01
      Discriminator masks -> 0x0
      Correlation scaler trigger delay 0x180 -> 0x5

      Analysis of R16_0 - FEE64 data synchronised
  356   Sun Sep 4 04:26:23 2016 TDSunday 4 September
14.20 R13 continues

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

      Merger errors messages for Q 19 - see attachment 4

     AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5 &
nnaida20 statistics - see attachments 6-16

     Analysis of R16_331 - FEE64 data synchronised - see attachment 17

aidas1> ./analyser2 /TapeData/Aug2016/R16_331
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  182657196 ( 1048839.0 Hz)
        Other data format:   78430804 (  450358.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         94 (       0.5 Hz)
                               RESUME:         95 (       0.5 Hz)
                              SYNC100:    1591884 (    9140.8 Hz)
                           FEE64 disc:   76825108 (  441138.8 Hz)
                             MBS info:      13623 (      78.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       4398 (      25.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      174.152 s

 FEE module #:   1 elapsed dead time       0.189 s
 FEE module #:   2 elapsed dead time       0.705 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       3.321 s
 FEE module #:   6 elapsed dead time       0.467 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.124 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.284 s
 FEE module #:  14 elapsed dead time       0.035 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.684 s
 FEE module #:  18 elapsed dead time       0.012 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.158 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   27.594s ( 1159.683 blocks/s,  72.480 Mb/s)


14.29 R16 stops


15.30 Noise measurements
      Found incorrect test pulse polarity connected to p+n and n+n FEE64s - corrected - see attachment 18

DSSSD #1 FEE64s, shaping time(us) versus pulser peak widths ch FWHM

      17  22  20  23 spectrum 1.8.L 
0.5  180 202 130 104
1    147 177 121 128
2    110 164 116 128
2.5   97 154 103 131
3.5   82 123  93 110
4.5   75 105  88 104
5.5   71  93  86  99
6     71  90  84  97
6.5   69  91  87  94
7     70  87  80  94
7.5   70  84  86  93
8     71  87  87  92
      
      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

16.26 R17 starts
      Pulser + 207Bi source

      ASIC settings 2016Sep02-18.17.01
      Discriminator masks -> 0-31 0x0 32-63 0x800000
      Correlation scaler trigger delay 0x180 -> 0x5
      Slow comparator 0x8

      Analysis of R17_0 - FEE64 data synchronised - see attachment 19

      top - attachment 20
      merger 1.9M items/s tapeserver 15Mb/s
  357   Sun Sep 4 04:35:42 2016 TDFEE64 & MACB configuration


                                                   UP
                                       nnaida1  nnaida9   nnaida17
LEFT                                   nnaida2  nnaida10  nnaida18                                   RIGHT
          nnaida23  nnaida15  nnaida7                               nnaida3   nnaida11  nnaida19
          nnaida24  nnaida16  nnaida8                               nnaida4   nnaida12  nnaida20
                                       nnaida5  nnaida13  nnaida21
                                       nnaida6  nnaida14  nnaida22
                                                  DOWN

                                           <LOOKING UPSTREAM>

LEFT-RIGHT n+n ohmic strips
UP-DOWN    p+n junction strips


    1     2     3     4     5     6     7     8     9    10    11    MACB NIM Module #   spare 1  spare 2

        <- #1 <- #2 <- #2 <- #2 <- #2 <- #1 <- #7 <- #7 <- #7 <- #7  Next

  -> #2 -> #3   6*   21     7    23   -> #8   1    17     3    19    Port 1
  -> #7 -> #4   5    22     8    24   -> #9   2    18     4    20    Port 2
        -> #5  13    29    15    31   -> #10  9    25    11    27    Port 3
        -> #6  14    30    16    32   -> #11 10    26    12    28    Port 4

    **    2     2     3     3     3     3     3     3     3     3    Mode
          y     y                                                    Rev B                        y

* = Master Timestamp FEE64

** MACB (rev A serial #13 mode 5) external 50MHz clock
   or
   MACB (rev B serial #11 mode 0) internal 50MHz cloxk

MACB HDMI cabling mechanically secured by 1U cable organiser above MACB NIM bin and by AIDA frame
- see attachments 1-3
  358   Mon Sep 5 06:54:02 2016 TDMonday 5 September
14.20 R17 continues

      DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

      No merger error messages

     AIDA SYNC, PAUSE, RESUME, Correlation L, SYNC time warp, Good Events, AIDA disc, nnaida6, nnaida5 &
nnaida20 statistics - see attachments 4-14

     Analysis of R17_584 - FEE64 data synaidas1> ./analyser2 /TapeData/Aug2016/R17_584
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258868072 ( 1733415.6 Hz)
        Other data format:    2219928 (   14864.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         94 (       0.6 Hz)
                               RESUME:         94 (       0.6 Hz)
                              SYNC100:    1365786 (    9145.5 Hz)
                           FEE64 disc:     842248 (    5639.8 Hz)
                             MBS info:      11706 (      78.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        400 (       2.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      149.340 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.026 s
 FEE module #:   3 elapsed dead time       0.007 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.439 s
 FEE module #:   6 elapsed dead time       0.329 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.104 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.001 s
 FEE module #:  13 elapsed dead time       0.515 s
 FEE module #:  14 elapsed dead time       0.290 s
 FEE module #:  15 elapsed dead time       0.078 s
 FEE module #:  16 elapsed dead time       0.036 s
 FEE module #:  17 elapsed dead time       0.379 s
 FEE module #:  18 elapsed dead time       0.044 s
 FEE module #:  19 elapsed dead time       0.122 s
 FEE module #:  20 elapsed dead time       0.063 s
 FEE module #:  21 elapsed dead time       0.218 s
 FEE module #:  22 elapsed dead time       0.467 s
 FEE module #:  23 elapsed dead time       0.232 s
 FEE module #:  24 elapsed dead time       0.059 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   26.172s ( 1222.687 blocks/s,  76.418 Mb/s)
chronised - see attachment 16

aidas1> ./analyser2 /TapeData/Aug2016/R17_584
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258868072 ( 1733415.6 Hz)
        Other data format:    2219928 (   14864.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         94 (       0.6 Hz)
                               RESUME:         94 (       0.6 Hz)
                              SYNC100:    1365786 (    9145.5 Hz)
                           FEE64 disc:     842248 (    5639.8 Hz)
                             MBS info:      11706 (      78.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        400 (       2.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:      149.340 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.026 s
 FEE module #:   3 elapsed dead time       0.007 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.439 s
 FEE module #:   6 elapsed dead time       0.329 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.104 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.001 s
 FEE module #:  13 elapsed dead time       0.515 s
 FEE module #:  14 elapsed dead time       0.290 s
 FEE module #:  15 elapsed dead time       0.078 s
 FEE module #:  16 elapsed dead time       0.036 s
 FEE module #:  17 elapsed dead time       0.379 s
 FEE module #:  18 elapsed dead time       0.044 s
 FEE module #:  19 elapsed dead time       0.122 s
 FEE module #:  20 elapsed dead time       0.063 s
 FEE module #:  21 elapsed dead time       0.218 s
 FEE module #:  22 elapsed dead time       0.467 s
 FEE module #:  23 elapsed dead time       0.232 s
 FEE module #:  24 elapsed dead time       0.059 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   26.172s ( 1222.687 blocks/s,  76.418 Mb/s)

15.08 R17 stops

Detector bias OFF
FEE64 power OFF 
USB-controlled ac mains relay OFF
Julabo FL11006 recirculating chiller OFF
ac mains cables to ac mains distributor disconnected
network cables aida1 and aida2 disconnected, rolled up and secured to stairwell rails

207Bi source removed and returned to B2F/F11 storage location
  363   Fri Sep 30 03:35:12 2016 TDFriday 30 September
11.37 Attempt to use external 50MHz clock for Master MACB (w/ 50MHz front panel label)

      NIM 50MHz -> NIM-TTL translator -> Master MACB (rear panel)

      Master MACB mode 0 -> 5

      Fails to ReSync, clock errors etc - see attachments 1-5

      Change NIM 50MHz -> 25MHz (to improve signal quality) and attempt ReSync (no power FEE64 power cycle) - no
change 

14.30 Master MACB mode -> 0

      NIM 25MHz -> Master MACB correlation clock input (front panel)

      BNC PB_4 frequency c. 26Hz
      Trigger output -> SCA -> LeCroy 429A -> Master MACB accepted trigger input

./analyser2 v /TapeData/Sept2016/R5_0 | grep MBS

 :
 :

 ***     MBS information: block:   471 ptr:  7203 data: 0x86807CAB module:  6 information type:  8 information
index:  0 MBS data:7CAB ts: 0x00000017AC2C4582
 ***     MBS information: block:   471 ptr:  7205 data: 0x8681575E module:  6 information type:  8 information
index:  1 MBS data:575E ts: 0x00000017AC2C4582
 ***     MBS information: block:   471 ptr:  7207 data: 0x86820000 module:  6 information type:  8 information
index:  2 MBS data:0000 ts: 0x00000017AC2C4582
 ***     MBS information: block:   471 ptr: 15767 data: 0x86800FA8 module:  6 information type:  8 information
index:  0 MBS data:0FA8 ts: 0x00000017AC669176
 ***     MBS information: block:   471 ptr: 15769 data: 0x8681576D module:  6 information type:  8 information
index:  1 MBS data:576D ts: 0x00000017AC669176
 ***     MBS information: block:   471 ptr: 15771 data: 0x86820000 module:  6 information type:  8 information
index:  2 MBS data:0000 ts: 0x00000017AC669176

0x576D0FA8 - 0x575E7CAB = 0xE92FD = 955133 * 40ns = 38205320ns = 38.2ms => 26Hz
0x00000017AC669176 - 0x00000017AC2C4582 = 0x3a4bf4 = 3820532 * 10ns = 38205320ns QED


15.45 Master MACB mode 0 -> 5
      
      NIM 50MHz -> Phillips 726 NIM-TTL-NIM -> Master MACB external clock (rear panel)
      Phillips 726 provides better quality TTL signal with c. 3ns rise/falltimes and 3.5V amplitude into 50 Ohm
cf. NIM-TTL-NIM module used previously
      See http://www.phillipsscientific.com/pdf/726ds.pdf

      RESET/SETUP DAQ OK 

      System-wide checks OK *except* nnaida19 fails ASIC timestamp check (no change)

      Start test run R6_0 to disk but no AIDA issue correlation triggers are recorded

      Can manually issue correlation triggers and resets via Correlation Status & Control tab

      Observe statistics count 3x for each trigger and reset - see https://elog.ph.ed.ac.uk/AIDA/362

aidas1> ./analyser2 v /TapeData/Sept2016/R6_0 | grep MBS

 :
 :
 Test #1 - manual correlation trigger

 ***     MBS information: block:  9626 ptr:  9991 data: 0x86807774 module:  6 information type:  8 information
index:  0 MBS data:7774 ts: 0x00000077517D2816
 ***     MBS information: block:  9626 ptr:  9993 data: 0x86817213 module:  6 information type:  8 information
index:  1 MBS data:7213 ts: 0x00000077517D2816
 ***     MBS information: block:  9626 ptr:  9995 data: 0x86820003 module:  6 information type:  8 information
index:  2 MBS data:0003 ts: 0x00000077517D2816

 :
 :
 Test #2 - manual correlation trigger x2 separated by ~10s

 ***     MBS information: block: 16751 ptr: 10787 data: 0x86807285 module:  6 information type:  8 information
index:  0 MBS data:7285 ts: 0x000000833335145A
 ***     MBS information: block: 16751 ptr: 10789 data: 0x86816A81 module:  6 information type:  8 information
index:  1 MBS data:6A81 ts: 0x000000833335145A
 ***     MBS information: block: 16751 ptr: 10791 data: 0x86820006 module:  6 information type:  8 information
index:  2 MBS data:0006 ts: 0x000000833335145A

 ***     MBS information: block: 16929 ptr:  8209 data: 0x86807BC5 module:  6 information type:  8 information
index:  0 MBS data:7BC5 ts: 0x000000837F39395A
 ***     MBS information: block: 16929 ptr:  8211 data: 0x86817D82 module:  6 information type:  8 information
index:  1 MBS data:7D82 ts: 0x000000837F39395A
 ***     MBS information: block: 16929 ptr:  8213 data: 0x86820006 module:  6 information type:  8 information
index:  2 MBS data:0006 ts: 0x000000837F39395A

0x67D827BC5 &#8722; 0x66A817285 = 0x13010940 = 318835008 × 20ns = 6376700160ns = 6.4s

0x837F39395A - 0x833335145A = 0x4C042500 = 1275340032 * 10ns = 12753400320ns = 12.8s => correlation clock
frequency = 25MHz *not* 50MHz

 :
 :
 Test #3 - manual correlation reset, followed by correlation trigger ~2s later 

 ***     MBS information: block: 17065 ptr: 15987 data: 0x86804D22 module:  6 information type:  8 information
index:  0 MBS data:4D22 ts: 0x00000083B96F05A6
 ***     MBS information: block: 17065 ptr: 15989 data: 0x86810344 module:  6 information type:  8 information
index:  1 MBS data:0344 ts: 0x00000083B96F05A6
 ***     MBS information: block: 17065 ptr: 15991 data: 0x86820000 module:  6 information type:  8 information
index:  2 MBS data:0000 ts: 0x00000083B96F05A6

0x3444d22 = 54807842 * 40ns = 2192313680ns = 2.2s

*re-check*

Correlation trigger input OK, pulse width 500ns
External clock is 50MHz
MACB master mode 6

So ... correlation clock frequency is 25MHz and Correlation DAQ Accepted Trigger not working

18.30 Per PCS request

      Master MACB clock output  25Mhz
                  reset output  not observed at DAQ GO as expected
  364   Fri Sep 30 07:52:56 2016 TDReport - low - AIDA correlation L/M/H statistics count 3x
The AIDA correlation L, M & H statistics count 3x for each correlation trigger,
i.e. counts and rates displayed by statistics are 3x actual counts and rates 
  367   Sat Oct 1 09:34:23 2016 TDInstalling latest (16 April 2015) MACB firmware
Installed Xilinx 14.7 LabTools + (free) WebPack license

Start iMPACT

Connect Xilinx programmer via 14-way ribbon cable to MACB PCB jtag connector
and connect USB cable to laptop

Powerup NIM bin with MACB (serial #13)

Auto-install Xilinx programmer driver

Use iMPACT Wizard to automagically find Xilinx device of MACB

Load new *.jed file
 Operations -> Program
 Operations -> Verify

TTY console log

Welcome to iMPACT
iMPACT Version: 14.7
Project: C:\Xilinx\14.7\LabTools\LabTools\auto_project.ipf created.
// *** BATCH CMD : setMode -bs
// *** BATCH CMD : setMode -bs
// *** BATCH CMD : setMode -bs
// *** BATCH CMD : setMode -bs
GUI --- Auto connect to cable...
// *** BATCH CMD : setCable -port auto
INFO:iMPACT - Digilent Plugin: Plugin Version: 2.4.4
INFO:iMPACT - Digilent Plugin: no JTAG device was found.
AutoDetecting cable. Please wait.
*** WARNING ***: When port is set to auto detect mode, cable speed is set to default 6 MHz regardless of explicit arguments supplied for 
setting the baud rates
PROGRESS_START - Starting Operation.
Connecting to cable (Usb Port - USB21).
Checking cable driver.
 Driver file xusb_xp2.sys found.
 Driver version: src=2301, dest=2301.
 Driver windrvr6.sys version = 10.2.1.0. WinDriver v10.21 Jungo (c) 1997 - 2010 Build Date: Aug 31 2010 x86_64 64bit SYS 14:14:44, 
version = 1021.
 Cable PID = 0008.
 Max current requested during enumeration is 300 mA.
Type = 0x0005.
write (count, cmdBuffer, dataBuffer) failed C0000004.
 Cable Type = 3, Revision = 0.
 Setting cable speed to 6 MHz.
Cable connection established.
Firmware version = 2301.
File version of C:/Xilinx/14.7/LabTools/LabTools/data/xusb_xp2.hex = 2401.
Firmware hex file version = 2401.
Downloading C:/Xilinx/14.7/LabTools/LabTools/data/xusb_xp2.hex.
Downloaded firmware version = 2401.
PLD file version = 200Dh.
 PLD version = 200Dh.
PROGRESS_END - End Operation.
Elapsed time =      1 sec.
Type = 0x0005.
ESN option: 000013E7C24E01.
Attempting to identify devices in the boundary-scan chain configuration...
INFO:iMPACT - Current time: 01/10/2016 09:30:42
// *** BATCH CMD : Identify -inferir 
PROGRESS_START - Starting Operation.
Identifying chain contents...'0': : Manufacturer's ID = Xilinx xc2c128, Version : 1
INFO:iMPACT:1777 - 
Reading C:/Xilinx/14.7/LabTools/LabTools/xbr/data/xc2c128.bsd...
INFO:iMPACT:501 - '1': Added Device xc2c128 successfully.
----------------------------------------------------------------------
----------------------------------------------------------------------
done.
PROGRESS_END - End Operation.
Elapsed time =      6 sec.
// *** BATCH CMD : identifyMPM 
// *** BATCH CMD : assignFile -p 1 -file "C:/Users/td/Local Documents/misc/macb_top_revB.jed"
'1': Loading file 'C:/Users/td/Local Documents/misc/macb_top_revB.jed' ...
done.
INFO:iMPACT:501 - '1': Added Device xc2c128 successfully.
----------------------------------------------------------------------
----------------------------------------------------------------------
INFO:iMPACT - Current time: 01/10/2016 09:32:54
// *** BATCH CMD : Program -p 1 -e -v 
PROGRESS_START - Starting Operation.
Maximum TCK operating frequency for this device chain: 33000000.
Validating chain...
Boundary-scan chain validated successfully.
'1': Erasing device...
'1': Erasure completed successfully.
'1': Programming device...
done.
'1': Verifying device...
done.
'1': Setting ISC done bits...done
'1': Programming completed successfully.
PROGRESS_END - End Operation.
Elapsed time =      2 sec.
INFO:iMPACT - Current time: 01/10/2016 09:33:01
// *** BATCH CMD : Verify -p 1 
PROGRESS_START - Starting Operation.
Maximum TCK operating frequency for this device chain: 33000000.
Validating chain...
Boundary-scan chain validated successfully.
'1': Verifying device...
done.
'1': Verification completed successfully.
PROGRESS_END - End Operation.
Elapsed time =      0 sec.
  368   Sun Oct 2 06:12:54 2016 TDSunday 2 October
14.15 DSSSD bias/leakage current OK - see attachments 1-2
      FEE64 temperatures OK - see attachment 3

      B2F/F11 temperature +27.5 deg C, d.p. +10.6 deg C, 34.3 RH %

14.24 R8 starts

      ASIC settings 2016Sep02-18.17.01

      Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
      Master MACB mode 5

      NIM 50MHz -> Phillips 726 NIM-TTL-NIM -> Master MACB - external clock (rear panel)

      Note Fuji Diamond Tool Ltd 1020 NIM-TTL level adaptor (NIM, pink front panel) with signal
      rise and falltimes quoted as <10ns produced poor quality 25/50MHz clock - the Phillips 726
      has significantly faster rise and falltimes and drices a higher amplitude signal to 50R load.

      BNC PB-4 trigger output -> Ortec SCA -> LeCroy 429A (500ns width) -> Master MACB - Correlation DAQ
Accepted Trigger

      BNC PB-4 frequency c. 26Hz

      Statistics - see attachments 4-14
      Master Timestamp - see attachment 15
      Correlation Status & Control - see attachment 16

      Successive correlation scaler events from R8_0

 ***     MBS information: block:   120 ptr:  7827 data: 0x868068B4 module:  6 information type:  8 information
index:  0 MBS data:68B4 ts: 0x00000019979ADAF8
 ***     MBS information: block:   120 ptr:  7829 data: 0x86812F00 module:  6 information type:  8 information
index:  1 MBS data:2F00 ts: 0x00000019979ADAF8
 ***     MBS information: block:   120 ptr:  7831 data: 0x86820000 module:  6 information type:  8 information
index:  2 MBS data:0000 ts: 0x00000019979ADAF8

 ***     MBS information: block:   126 ptr:  7005 data: 0x8680FDF4 module:  6 information type:  8 information
index:  0 MBS data:FDF4 ts: 0x0000001997D52FF8
 ***     MBS information: block:   126 ptr:  7007 data: 0x86812F0E module:  6 information type:  8 information
index:  1 MBS data:2F0E ts: 0x0000001997D52FF8
 ***     MBS information: block:   126 ptr:  7009 data: 0x86820000 module:  6 information type:  8 information
index:  2 MBS data:0000 ts: 0x0000001997D52FF8

0x2F0EFDF4 - 0x2F0068B4 = 0xE9540 = 955712 * 40ns (assume 25MHz) = 38228480ns = 38.2ms

0x1997D52FF8 - 0x19979ADAF8 = 0x3A5500 = 3822848 * 10ns = 38228480ns = 38.2ms

=> correlation clock frequency = 25MHz

Master MACB (serial #13) mode = 5, 50MHz clock to external clock input (rear panel)
       clock output                             25MHz
       reset output                             reset not observed at DAQ GO
                                                reset observed using Correlation status & control tab -> Force
Correlation Reset
       reset request output                     not tested
       correlation DAQ accepted trigger output  c.26Hz

Could not persuade MACB serial #13 to operate in mode 0, external clock input (rear panel) disconnected) 
Persistent failures to ReSYNC by nnaida11, 12, 17, 18, 19, & 20 (see https://elog.ph.ed.ac.uk/AIDA/357)
following RESET/SETUP, FEE64 power cycle, MACB NIM bin power cycle and re-seating master MACB HDMI cables

Switch back to previous Master MACB serial #11 

Master MACB (serial #11) mode = 0
       clock output                             25MHz
       reset output                             reset not observed at DAQ GO
                                                reset not observed using Correlation status & control tab -> Force
Correlation Reset
       reset request output                     not tested
       correlation DAQ accepted trigger output  not observed
  372   Thu Oct 6 12:39:14 2016 TDImages of setup at F11
  373   Thu Oct 6 13:09:50 2016 TDImages of setup at F11 - 2
  375   Thu Oct 6 13:20:34 2016 TDImages of setup at F11 - 3
  377   Thu Oct 6 13:22:24 2016 TDImages of setup at F11 - 4
  387   Sat Oct 15 03:27:31 2016 TDSaturday 15 October
11.27 DAQ found stopped
      Most recent file R42_164, last accessed 07.18
      Restart merger OK
      DAQ STOP OK

      System-wide checks - see attachments 1-5
      ASIC clock timestamp, SYNC errors fail *but* SYNC pulses OK

      ReSYNC OK
      System-wide checks OK

      DSSSD bias & leakage currents OK - see attachments 6 & 7
      FEE64 temperatures OK - see attachment 8

      R42_164 ADC data synchronised

aidas1> ./analyser2 /TapeData/Sept2016/R42_164
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:       2444
          ADC data format:   19809141 ( 1404089.4 Hz)
        Other data format:     131455 (    9317.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          7 (       0.5 Hz)
                               RESUME:          7 (       0.5 Hz)
                              SYNC100:     129011 (    9144.4 Hz)
                           FEE64 disc:       2010 (     142.5 Hz)
                             MBS info:        420 (      29.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:         21 (       1.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:       14.108 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.000 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.334 s
 FEE module #:   6 elapsed dead time       0.042 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.000 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.000 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.000 s
 FEE module #:  18 elapsed dead time       0.005 s
 FEE module #:  19 elapsed dead time       0.000 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.000 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:    0.586s ( 4171.093 blocks/s, 260.693 Mb/s)

11.39 DAQ starts (R43)

12.07 Disk housekeeping caused TapeServer problem
      DAQ stopped OK after some minutes and restarting merger

12.43 DAQ starts (R46)

18.26 DAQ stalled
      Latest file R46_109 (most recent access 18.26)
      Merger error messages - see attachment 9
      Restart merger - error messages - see attachment 10
   
      Analysis of R46_109

      ADC data synchronised

aidas1> ./analyser2 /TapeData/Sept2016/R46_109
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:       4688
          ADC data format:   37985730 ( 1342216.8 Hz)
        Other data format:     263662 (    9316.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         25 (       0.9 Hz)
                               RESUME:         25 (       0.9 Hz)
                              SYNC100:     258788 (    9144.2 Hz)
                           FEE64 disc:       3981 (     140.7 Hz)
                             MBS info:        843 (      29.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:         41 (       1.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:       28.301 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.013 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.298 s
 FEE module #:   6 elapsed dead time       0.200 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.001 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.003 s
 FEE module #:  15 elapsed dead time       0.000 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.086 s
 FEE module #:  18 elapsed dead time       0.054 s
 FEE module #:  19 elapsed dead time       0.097 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.001 s
 FEE module #:  23 elapsed dead time       0.000 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:    2.102s ( 2230.721 blocks/s, 139.420 Mb/s)


11.00 Sunday 16 October 
      DAQ STOP requested - nnaida1-24 all stop *except* nnaida6 which remains going - Run Control displays RESET
only
      DAQ RESET
      System-wide checks all OK
      Normal setup and DAQ restart (see https://elog.ph.ed.ac.uk/AIDA/388 )
      DAQ SETUP
      
  388   Sun Oct 16 03:14:41 2016 TDSunday 16 October
11.12 DAQ start (R47)

      Detector bias and leakage currents OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3

      Statistics - see attachments 4-14

11.56 Temperature +25.0 deg C, d.p. +8.4 deg C, RH 34.5%

20.33 DAQ found stopped
      Most recent file R47_120 (last access 18.26)

      R47_120 ADC data synchronised

      Restart merger and request DAQ STOP - DAQ stops OK

      System-wide tests SYNC errors and SYNC pulses - see attachments 15 & 16 - all other tests OK

      DAQ RESET & SETUP OK

      Normal restart which failed shortly after enabling ASIC readout - restarting merger did not
      recover DAQ control - power cycle required

21.05 DAQ starts (R49)
      
  390   Mon Oct 17 04:10:30 2016 TDMonday 17 October
12.05 DAQ found stopped
      Most recent file R49_226, last accessed 06.26
      Restart merger OK
      DAQ STOP OK

      System-wide checks - see attachments 1-2 
      SYNC errors and SYNC pulses fail

      ReSYNC OK
      System-wide checks OK
      Restart merger & tapeserver

      DSSSD bias & leakage currents OK - see attachments 3 & 4
      FEE64 temperatures OK - see attachment 5

      DAQ start
      #1 nnaida15 data transfer #1 tag drops at DAQ GO
         disable H & data transfer #1
         restart merger & tapeserver
         enable H & data transfer #1

12.22 #2 DAQ start (R51) OK

20.33 DAQ found stopped
      Most recent file R51_145, last accessed 18.30
      Restart merger OK
      DAQ STOP OK

      System-wide checks - see attachments 6 & 7
      SYNC errors and SYNC pulses fail

      DAQ RESET & SETUP OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

20.41 DAQ GO (R52)
  391   Tue Oct 18 01:43:04 2016 TDTuesday 18 October
09/42 DAQ found stopped, no merger error messages
      Most recent file R52_211, last accessed 06.30
      Restart merger OK
      DAQ STOP OK

aidas1> ./analyser2 /TapeData/Sept2016/R52_211
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:       7619
          ADC data format:   61809254 ( 1624588.3 Hz)
        Other data format:     354167 (    9308.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         28 (       0.7 Hz)
                               RESUME:         28 (       0.7 Hz)
                              SYNC100:     347607 (    9136.5 Hz)
                           FEE64 disc:       5364 (     141.0 Hz)
                             MBS info:       1140 (      30.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:         88 (       2.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:       38.046 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.011 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.058 s
 FEE module #:   6 elapsed dead time       0.165 s
 FEE module #:   7 elapsed dead time       0.000 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.000 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.002 s
 FEE module #:  15 elapsed dead time       0.055 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.057 s
 FEE module #:  18 elapsed dead time       0.089 s
 FEE module #:  19 elapsed dead time       1.021 s
 FEE module #:  20 elapsed dead time       0.000 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.000 s
 FEE module #:  23 elapsed dead time       0.085 s
 FEE module #:  24 elapsed dead time       0.004 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:    1.793s ( 4249.377 blocks/s, 265.586 Mb/s)

      R51_211 ADC data synchronised

      DSSSD bias & leakage currents OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3

      System-wide checks - see attachments 4 & 5 
      SYNC errors and SYNC pulses fail, all other tests OK

      Statistics - see attachments 6-16

      DAQ RESET & SETUP OK
      ReSYNC OK
      System-wide checks OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

10.02 DAQ start (R53)

18.20 DAQ OK

      System-wide checks *all* OK

      Statistics - see attachments 17-27

18.30 DAQ stalled on schedule 
      Most recent file R53_193 last accessed 18.30
      AIDA SYNC rates zero in large fraction of FEE64s - see attachment 28
      SYNC errors and SYNC pulses fail - see attachments 29-32

      DAQ STOP
      DAQ RESET
      Restart merger
      DAQ RESET OK
      DAQ SETUP OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

      System-wide checks *all* OK

18.45 DAQ starts (R54)
      merging stops shortly after start of run - see attachment 33
 
      DAQ HALT, RESET, SETUP OK
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

18.52 DAQ starts (R55)
  393   Tue Oct 18 06:56:52 2016 TDDAQ status
From the installation of the RG58 cable for the 50MHz TTL clock from BRIKEN DAQ
to AIDA Master MACB external clock input we observe DAQ stalls at c. 06.30 and
18.30 daily. This coincides with the nominal start times of the ORNL Clover detector
LN2 fill cycles.

run          start stop  files   recovery

AIDA Master MACB internal clock

R1   28.9.16 15.02 11.20 R1_304  OK

RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input

     9.10.16 16.55 17.30         no data transfer, TS_SYNC phase = 0, OK
     9.10.16 17.40 19.15 est     no data transfer, TS_SYNC phase = 0, ReSYNC
     9.10.16 22.20 01.40         no data transfer, TS_SYNC phase = 1, ?

    10.10.16 14.20 16.50         no data transfer, TS_SYNC phase = 2, ?
    10.10.16 18.35 18.40/19.40   no data transfer, TS_SYNC phase = 3, some FEE64s stopped later than others, ReSYNC
    10.10.16 20.36 06.35         no data transfer, TS_SYNC phase = 0, power cycle


RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input
replaced by RG58 cable

run          start stop  files   recovery

R17 11.10.16 15.46 08.22 R17_440 soft reset

R21 12.10.16 14.05 15.30         power dip - power cycle
R22 12.10.16 16.20 19.05         OK 

R24 13.10.16                     OK
R25 13.10.16                     OK
R26 13.10.16                     OK
R27 13.10.16                     OK
R28 13.10.16                     OK
R29 13.10.16                     OK

R32 13.10.16 21.20 22.34 R32_26  soft reset
R33 14.10.16 00.01 00.25 ?       power cycle

R40 14.10.16 01.49 06.26 R40_91  soft reset 
R41 14.10.16                     OK
R42 15.10.16 22.45 07.18 R42_164 System-wide tests - SYNC errors fail *but* SYNC pulses OK - ReSYNC only 
R43 15.10.16 11.39 12.43         disk housekeeping caused merger/tapeserver problem

R46 15.10.16 12.43 18.26 R46_109 soft reset
R47 16.10.16 11.12 18.26 R47_120 power cycle

R49 16.10.16 21.05 06.26 R49_226 soft reset

R51 17.10.16 12.21 18.30 R51_145 soft reset
R52 17.10.16 20.41 06.30 R52_211 soft reset
R53 18.10.16 10.02 18.30 R53_193 soft reset

R55 18.10.16 18.52 19.17 R55_10  soft reset

R60 19.10.16 09.35 16.28 R60_169 OK
R61 19.10.16 16.30 18.26 R61_69  power cycle

R63 19.10.16 20.05 06.46 R63_291 soft reset
R64 20.10.16 09.45 18.25 R64_240 OK - System-wide tests - SYNC errors fail & SYNC pulses fail - other checks OK
R65 20.10.16 18.49 19.17 R65_12  power cycle            
R66 20.10.16 20.02 10.55 R66_396 soft reset  auto LN2 fill may have failed, manually selected c. 11.00
R67 21.10.16 14.02 18.26 R67_121 soft reset
R68 21.10.16 18.40 18.44 R68_1   soft reset

R70 21.10.16 18.53 06.46 R70_315 soft reset
R71 22.10.16 10.42 18.25         OK

R73 22.10.16 19.07 19.17 R73_4   soft reset
R74 22.10.16 19.38 06.31 R74_296 soft reset

R76 23.10.16 10.46 11.52 R76_29  merger crashed, soft reset
R77 23.10.16 18.50 19.17         disk housekeeping caused merger/tapeserver problem
R78 23.10.16 19.47 06.46 R78_291 soft reset
R79 24.10.16 11.07 18.46 R79_182 soft reset
R80 24.10.16 18.56 19.17 R80_5   DAQ stall, soft reset
R81 24.10.16 19.29 06.26 R81_169 soft reset
R82 25.10.16 10.22 18.30 R82_125 -> safe state

R83 27.10.16 14.35 18.30 R83_125 soft reset
R86 27.10.16 20.48 06.26 R86_245 soft reset
R87 28.10.16 14.34 18.47 R87_274 soft reset

R89 29.10.16 19.10 07.17 R89_102 power cycle

soft reset => no FEE64 power cycle
      Restart merger
      DAQ STOP
      System wide checks 
      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1
  394   Wed Oct 19 01:14:10 2016 TDWednesday 19 October
09.13 DAQ found stopped, no merger error messages
      Most recent file R55_10, last accessed 19.17
      Restart merger OK
      DAQ STOP OK

aidas1> ./analyser2 /TapeData/Sept2016/R55_10
 *** GREAT format 3.2.0 analyser - TD - May 2014
 *** ERROR: READ I/O error:       5002
                   blocks:      23037
          ADC data format:  187091367 ( 2006959.5 Hz)
        Other data format:     867516 (    9306.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         62 (       0.7 Hz)
                               RESUME:         62 (       0.7 Hz)
                              SYNC100:     851796 (    9137.4 Hz)
                           FEE64 disc:      12836 (     137.7 Hz)
                             MBS info:       2760 (      29.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        124 (       1.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 Timestamp elapsed time:       93.221 s
 FEE module #:   1 elapsed dead time       0.000 s
 FEE module #:   2 elapsed dead time       0.000 s
 FEE module #:   3 elapsed dead time       0.000 s
 FEE module #:   4 elapsed dead time       0.000 s
 FEE module #:   5 elapsed dead time       0.922 s
 FEE module #:   6 elapsed dead time       1.113 s
 FEE module #:   7 elapsed dead time       0.017 s
 FEE module #:   8 elapsed dead time       0.000 s
 FEE module #:   9 elapsed dead time       0.001 s
 FEE module #:  10 elapsed dead time       0.000 s
 FEE module #:  11 elapsed dead time       0.000 s
 FEE module #:  12 elapsed dead time       0.000 s
 FEE module #:  13 elapsed dead time       0.000 s
 FEE module #:  14 elapsed dead time       0.005 s
 FEE module #:  15 elapsed dead time       0.016 s
 FEE module #:  16 elapsed dead time       0.000 s
 FEE module #:  17 elapsed dead time       0.446 s
 FEE module #:  18 elapsed dead time       0.435 s
 FEE module #:  19 elapsed dead time       0.925 s
 FEE module #:  20 elapsed dead time       0.015 s
 FEE module #:  21 elapsed dead time       0.000 s
 FEE module #:  22 elapsed dead time       0.000 s
 FEE module #:  23 elapsed dead time       0.221 s
 FEE module #:  24 elapsed dead time       0.000 s
 FEE module #:  25 elapsed dead time       0.000 s
 FEE module #:  26 elapsed dead time       0.000 s
 FEE module #:  27 elapsed dead time       0.000 s
 FEE module #:  28 elapsed dead time       0.000 s
 FEE module #:  29 elapsed dead time       0.000 s
 FEE module #:  30 elapsed dead time       0.000 s
 FEE module #:  31 elapsed dead time       0.000 s
 FEE module #:  32 elapsed dead time       0.000 s

 *** Program elapsed time:   22.266s ( 1034.644 blocks/s,  64.665 Mb/s)

      R55_10 ADC data synchronised

      DSSSD bias & leakage currents OK - see attachments 1 & 2
      FEE64 temperatures OK - see attachment 3

      SYNC statistics, System Wide Checks SYNC error & SYNC pulses fail - see attachments 4-8

      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

      #1 DAQ start (R58)
      nnaida7 data transfer #1 tag (X) drops at DAQ GO
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1  

      #2 DAQ start (R59)
      nnaida15 data transfer #1 tag (X) drops at DAQ GO
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1  

09.35 DAQ start (R60)

16.28 DAQ stop OK
      Final file R60_169

16.30 DAQ start (R61)
      BNC PB-4 pulser OFF

      Background run - files R61_0 - R61_31

      + 60Co (ID# 0115) - files R61_33 - R61_49 

      60Co source placed on top of AIDA snout ~ above DSSSD #3

      DAQ stalled at 18.28 (during R61_49) - as expected! See https://elog.ph.ed.ac.uk/AIDA/393

      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1

20.05 DAQ start (R63)
      BNC PB-4 pulser ON, 60Co source removed
ELOG V3.1.3-7933898