AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
ID Date Author Subject
  701   Tue Mar 11 11:53:18 2025 TDMonday 10 March
18.30 AIDA powered up for brief test

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
       ASIC settings at power up - slow comparator 0x64

      per p+n FEE64 1.8.L spectra - attachment 4
       pulser peak width aida9 55 ch FWHM (39keV FWHm), aida15 53 ch FWHM (37keV FWHM)

      p+n FEE64 aida15 1.8.W spectra - 20us FSR - attachments 5-6

      saved *.L, *.H, *.W spectra for aida15 - attachments 7-11
       *.W spectra - 20us, 200us, 2ms & 20ms FSR 


19.27 DAQ STOP
      DSSSD bias OFF
      FEE64 power OFF
Attachment 1: Screenshot_from_2025-03-10_18-41-32.png
Screenshot_from_2025-03-10_18-41-32.png
Attachment 2: Screenshot_from_2025-03-10_18-42-02.png
Screenshot_from_2025-03-10_18-42-02.png
Attachment 3: Screenshot_from_2025-03-10_18-42-33.png
Screenshot_from_2025-03-10_18-42-33.png
Attachment 4: Screenshot_from_2025-03-10_18-43-07.png
Screenshot_from_2025-03-10_18-43-07.png
Attachment 5: Screenshot_from_2025-03-10_18-48-03.png
Screenshot_from_2025-03-10_18-48-03.png
Attachment 6: Screenshot_from_2025-03-10_18-48-39.png
Screenshot_from_2025-03-10_18-48-39.png
Attachment 7: 2025Mar10-17.59.58.tar.gz
Attachment 8: 2025Mar10-18.05.01.tar.gz
Attachment 9: 2025Mar10-18.06.29.tar.gz
Attachment 10: 2025Mar10-18.12.03.tar.gz
Attachment 11: 2025Mar10-18.17.55.tar.gz
  700   Sat Feb 22 12:35:38 2025 TDOffline analysis data files R9_85-R9_199 (84Mo setting)
Start 00:43 22.2.25 https://elog.gsi.de/despec/G-24-00302/57
AIDA data file R9_85

Converting hexadecimal timestamp to decimal: 1740181266259754800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Friday, February 21, 2025 11:41:06.259 PM
Your time zone: Friday, February 21, 2025 11:41:06.259 PM GMT+00:00


End 12:56 22.2.25 https://elog.gsi.de/despec/G-24-00302/85 (start of next degrader setting)
AIDA data file R9_199

Converting hexadecimal timestamp to decimal: 1740224934731157800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Saturday, February 22, 2025 11:48:54.731 AM
Your time zone: Saturday, February 22, 2025 11:48:54.731 AM GMT+00:00

Analysis data files FEB25/R9_85 and FEB25/R9_199
 - no timewarps
 - deadtime aida05 c. 37% and 39% due to high SC41 scaler input to MACB - issue subsequently fixed 
 - deadtime aida04 c. 4%, all other FEE64s <<1%




FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4

n+n Ohmic FEE64s 2, 4

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (507 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   151keV < LEC energy < 1000keV
    to select candidate beta events
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0
   (x,y) strips corresponding to maximum energy
   p+n junction and n+n Ohmic side HEC 

- HEC veto 
   not available - only 1x AIDA DSSSD installed

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us





per FEE64 LEC ADC data items 268ms/channel - attachments 3-5
- all
- 150keV < energy < 1500keV
- energy > 1500keV

absence of high instantaneous rates on spill cf. S181, S100, S505 etc
spill structure visible for energies > 1500keV - probably to be expected as majority of such events should be light ions

per FEE64 HEC ADC data items 268ms/channel - attachments 6-8

- all
- 100MeV < energy < 1000MeV
- energy > 1000MeV

Implant & decay event rates 262us/channel - attachment 9

LEC m_p versus m_n - attachment 10

LEC e_p versus e_n - 20keV/channel - attachment 11

HEC x strip versus y strip - attachment 12

- HEC-LEC implant decay time difference <1s
- HEC-LEC implant decay time difference <100s

shows x-y window used to select 82Nb events

HEC m_p versus m_n - attachment 13

HEC e_p versus e_n - 20MeV/channel - attachment 14


HEC-LEC implant-decay time difference (4ms/channel) versus - attachment 15
- HEC energy 
- LEC energy
- x & y strip 

x strips 130 & 136 disabled

HEC & LEC p strip - n strip time difference (2us/channel, offset=2000 channels) - attachment 16

HEC E (20MeV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 17

LEC E (20keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 18

LEC e_p - e_n (5.6keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 19

HEC x & y strip versus implant-decay time difference (4.194ms/channel) - attachment 20

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 21
- blue - forward
- cyan - backward

Note
- no FRS PID
- no HEC dE
- no HEC veto
- no HEC energy gates - assume all events within x-y window *stop* in AIDA DSSSD - 82Nb events known to overlap in x-y with (longer-lived, more numerous) 83Nb events
- no front-back time difference
- no clustering
- *all* LEC channels with valid ADC offsets (507 of 512) included *except* 2x strips 130 & 136

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 22

Weighted least squares fit channels 1-12 half life 55(9)ms cf. NNDC 50.0(3)ms.
Attachment 1: R9_85.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  143503687 (  365856.8 Hz)
        Other data format:  118416315 (  301897.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2560 (       6.5 Hz)
                               RESUME:       2560 (       6.5 Hz)
                              SYNC100:      33461 (      85.3 Hz)
                              WR48-63:      33461 (      85.3 Hz)
                           FEE64 disc:     326756 (     833.1 Hz)
                             MBS info:  118017517 (  300880.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      29285 (      74.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      392.240 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.069                0.000
  2                0.000                0.000
  3               17.209                0.000
  4              146.817                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.009                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7002448      45530          0          0          0          0        930        930      31901      11769          0       2888
  1   22874309      77583          0          0         18         18       3036       3036      71475          0          0       7512
  2    6498360    4555402          0          0          0          0       1475       1475      34629    4517823          0       3971
  3   82625693      69682          0          0       1306       1306      10819      10819      45432          0          0       3669
  4    4358474  109462549          0          0       1235       1235      14045      14045       2695  109429294          0        206
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6         88          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8    2927664      19415          0          0          0          0        391        391      18633          0          0       4389
  9   14346055    3938106          0          0          1          1       2392       2392      18452    3914868          0       1270
 10        128          0          0          0          0          0          0          0          0          0          0          0
 11    1956985     169485          0          0          0          0        257        257      25208     143763          0       3888
 12        212          0          0          0          0          0          0          0          0          0          0          0
 13         55          0          0          0          0          0          0          0          0          0          0          0
 14     913148      78563          0          0          0          0        116        116      78331          0          0       1492
 15         68          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   24.117s ( 1326.854 blocks/s,  82.928 Mb/s)
Attachment 2: R9_199.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  154723223 (  435714.6 Hz)
        Other data format:  107196777 (  301875.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2462 (       6.9 Hz)
                               RESUME:       2462 (       6.9 Hz)
                              SYNC100:      33323 (      93.8 Hz)
                              WR48-63:      33323 (      93.8 Hz)
                           FEE64 disc:     288355 (     812.0 Hz)
                             MBS info:  106836852 (  300862.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      32530 (      91.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      355.102 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.008                0.000
  1                0.147                0.000
  2                0.000                0.000
  3               21.597                0.000
  4              136.749                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.006                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9317141      49848          0          0          6          6       1239       1239      36708      10650          0       3256
  1   27128444      89326          0          0         31         31       3596       3596      82072          0          0       8340
  2    7611251    4365052          0          0          0          0       1569       1569      37972    4323942          0       4609
  3   84626002      75259          0          0       1316       1316      10898      10898      50831          0          0       3978
  4    4974948   98704420          0          0       1107       1107      12783      12783       2668   98673972          0        187
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6        122          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8    3383262      22113          0          0          0          0        436        436      21241          0          0       4811
  9   14397933    3689217          0          0          2          2       2371       2371      21039    3663432          0       1355
 10          1          0          0          0          0          0          0          0          0          0          0          0
 11    2194427     194484          0          0          0          0        298        298      29032     164856          0       4304
 12        140          0          0          0          0          0          0          0          0          0          0          0
 13        260          0          0          0          0          0          0          0          0          0          0          0
 14    1088899       7058          0          0          0          0        133        133       6792          0          0       1690
 15        393          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.438s ( 1210.402 blocks/s,  75.650 Mb/s)
Attachment 3: Screenshot_from_2025-03-05_10-04-30.png
Screenshot_from_2025-03-05_10-04-30.png
Attachment 4: Screenshot_from_2025-03-05_10-04-55.png
Screenshot_from_2025-03-05_10-04-55.png
Attachment 5: Screenshot_from_2025-03-05_10-05-29.png
Screenshot_from_2025-03-05_10-05-29.png
Attachment 6: Screenshot_from_2025-03-05_10-07-35.png
Screenshot_from_2025-03-05_10-07-35.png
Attachment 7: Screenshot_from_2025-03-05_10-07-56.png
Screenshot_from_2025-03-05_10-07-56.png
Attachment 8: Screenshot_from_2025-03-05_10-08-29.png
Screenshot_from_2025-03-05_10-08-29.png
Attachment 9: Screenshot_from_2025-03-05_10-09-24.png
Screenshot_from_2025-03-05_10-09-24.png
Attachment 10: Screenshot_from_2025-03-05_10-10-29.png
Screenshot_from_2025-03-05_10-10-29.png
Attachment 11: Screenshot_from_2025-03-05_10-11-17.png
Screenshot_from_2025-03-05_10-11-17.png
Attachment 12: Screenshot_from_2025-03-05_10-12-05.png
Screenshot_from_2025-03-05_10-12-05.png
Attachment 13: Screenshot_from_2025-03-05_10-12-35.png
Screenshot_from_2025-03-05_10-12-35.png
Attachment 14: Screenshot_from_2025-03-05_10-13-38.png
Screenshot_from_2025-03-05_10-13-38.png
Attachment 15: Screenshot_from_2025-03-05_10-14-59.png
Screenshot_from_2025-03-05_10-14-59.png
Attachment 16: Screenshot_from_2025-03-05_10-15-51.png
Screenshot_from_2025-03-05_10-15-51.png
Attachment 17: Screenshot_from_2025-03-08_09-11-40.png
Screenshot_from_2025-03-08_09-11-40.png
Attachment 18: Screenshot_from_2025-03-08_09-11-58.png
Screenshot_from_2025-03-08_09-11-58.png
Attachment 19: Screenshot_from_2025-03-08_09-13-07.png
Screenshot_from_2025-03-08_09-13-07.png
Attachment 20: Screenshot_from_2025-03-08_14-01-50.png
Screenshot_from_2025-03-08_14-01-50.png
Attachment 21: Screenshot_from_2025-03-08_12-21-22.png
Screenshot_from_2025-03-08_12-21-22.png
Attachment 22: 82Nb.png
82Nb.png
  699   Thu Feb 20 17:06:42 2025 JB, CC, TD,MPWR error for aida 13,14,15,16

WR timestamp errors resolved after reseating the HDMI cables to the MACB

  698   Wed Feb 12 10:46:01 2025 TDOffline analysis data files S181 R4_351-396
Analysis data files R4_351-396


Data file R4_351 first WR ts 0x17D7B4C72B0BA9C8
Converting hexadecimal timestamp to decimal: 1718040550378809900
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 5:29:10.378 PM
Your time zone: Monday, June 10, 2024 6:29:10.378 PM GMT+01:00 DST
Relative: 8 months ago

Data file R4_396 first WR ts 0x17D7BE86408888E8
Converting hexadecimal timestamp to decimal: 1718051266682718500
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 8:27:46.682 PM
Your time zone: Monday, June 10, 2024 9:27:46.682 PM GMT+01:00 DST
Relative: 8 months ago

Attachments 1-2 - analysis data files R4_351 and R4_396
 max. *time averaged* deadtime FEE64 #7 (aida08) 1.7% and 1.7% respectively
 all other FEE64 deadtimes < 1%





FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 11200keV (wide open for first pass analysis)

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   LEC energy > 151keV
    to select candidate beta  and alpha events - will include light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us




Number of events observed

 *** scaler # 1 count:  35941696 DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 2 count:  31960753 DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 3 count:   1499844 DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 4 count:   1297356 DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 5 count:   5446969 DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler # 6 count:  75030747 DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8

DSSSD#1 implant events
x=m_p=0 ?
y=m_n=0 ?

DSSSD#2 implant events
x=m_p=0 ?
y=m_n=0 ?


Attachments 3-6 - per FEE64 LEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV

Some deadtime observed on spill, none observed off spill. No significant variation in per FEE64 LEC data rates.

Attachments 7-8 - per FEE64 LEC hit patterns
 - 150keV < energy < 1500keV
 - energy > 1500keV

Aside from usual hot channels at cable/DSSSD boundaries good hit pattern observed. For > 1500keV events observe flat field in x-plane, focussed in y-plane.

Attachments 9-11 - per FEE64 HEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Note hot HEC channel in aida08 - should be disabled.

Attachment 12 - per DSSSD decay and implant rates 262us/channel - common x and y scales

Attachment 13 - per DSSSD LEC m_p versus m_n 
 - no conditions

Attachment 14 - per DSSSD LEC p strip versus n strip
 - no conditions

Attachments 15-16 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)

Attachments 17-18 - per DSSSD p strip versus n strip
 - HEC-LEC time difference <1s
 - HEC-LEC time difference <100s

Attachment 19 -  per DSSSD HEC m_p versus m_n

Attachment 20 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 21 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 22 - DSSSD#1 HEC E_p versus DSSSD#2 HEC E_p - x and y axes 20MeV/channel
 - few events stop in DSSSD#2
 - most events lower Z and A - fission fragments?

Attachment 23 - per DSSSD per pixel HEC-LEC time difference 4.096us/channel
 - events observed to <<100us

Attachment 24 - per DSSSD per pixel HEC-LEC time difference (1s/channel) versus LEC energy (20keV/channel)
 - alpha events long lived 
 - some evidence of effect of on spill deadtime in implant-decay correlations (cf. S100 and S505)

Attachment 25 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events as expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle
Attachment 1: R4_396.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256709815 ( 1065042.5 Hz)
        Other data format:    5210185 (   21616.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        653 (       2.7 Hz)
                               RESUME:        653 (       2.7 Hz)
                              SYNC100:      32897 (     136.5 Hz)
                              WR48-63:      32897 (     136.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5143085 (   21337.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     639531 (    2653.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      241.032 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.017                0.000
  2                0.000                0.000
  3                0.848                0.000
  4                0.000                0.000
  5                0.227                0.000
  6                0.000                0.000
  7                4.199                0.000
  8                0.000                0.000
  9                1.796                0.000
 10                0.020                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.823                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3315802       8090          0          0          0          0        430        430          0       7230          0       9958
  1    8650533       2148          0          0          3          3       1071       1071          0          0          0      24316
  2    7128165    1944546          0          0          0          0       1082       1082          0    1942382          0      12931
  3   24378546       6508          0          0        152        152       3102       3102          0          0          0      19563
  4    2929004     400994          0          0          0          0        424        424          0     400146          0      12707
  5   19313956       4950          0          0         45         45       2430       2430          0          0          0      22052
  6    5646509     333129          0          0          0          0        723        723          0     331683          0      12637
  7   62853684      16350          0          0        195        195       7980       7980          0          0          0     436912
  8    8105988       2098          0          0          0          0       1049       1049          0          0          0      14497
  9   45373582    1939538          0          0        183        183       6118       6118          0    1926936          0      14193
 10   14519870       3442          0          0         13         13       1708       1708          0          0          0       8558
 11    2400015     535424          0          0          0          0        358        358          0     534708          0      14419
 12    4828613       1128          0          0          0          0        564        564          0          0          0       8079
 13    7222020       1834          0          0          0          0        917        917          0          0          0      12852
 14    3474619        796          0          0          0          0        398        398          0          0          0       8899
 15   36568909       9210          0          0         62         62       4543       4543          0          0          0       6958
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.098s ( 1226.164 blocks/s,  76.635 Mb/s)
Attachment 2: R4_351.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256848692 ( 1048248.2 Hz)
        Other data format:    5071308 (   20697.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        624 (       2.5 Hz)
                               RESUME:        624 (       2.5 Hz)
                              SYNC100:      32913 (     134.3 Hz)
                              WR48-63:      32913 (     134.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5004234 (   20423.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     629529 (    2569.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      245.027 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.032                0.000
  2                0.000                0.000
  3                0.744                0.000
  4                0.000                0.000
  5                0.181                0.000
  6                0.000                0.000
  7                4.062                0.000
  8                0.000                0.000
  9                1.739                0.000
 10                0.001                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.087                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3260523       8194          0          0          0          0        422        422          0       7350          0       9200
  1    8620748       2224          0          0          6          6       1106       1106          0          0          0      21909
  2    7063739    1926312          0          0          0          0       1116       1116          0    1924080          0      12003
  3   24732934       6660          0          0        134        134       3196       3196          0          0          0      17638
  4    2925688     370692          0          0          0          0        405        405          0     369882          0      11429
  5   18974781       4842          0          0         34         34       2387       2387          0          0          0      20060
  6    5599469     300187          0          0          0          0        782        782          0     298623          0      11800
  7   63229156      16356          0          0        190        190       7988       7988          0          0          0     445010
  8    7941245       2048          0          0          0          0       1024       1024          0          0          0      13314
  9   45626532    1921841          0          0        173        173       5873       5873          0    1909749          0      13056
 10   14656402       3672          0          0          3          3       1833       1833          0          0          0       7698
 11    2320422     495242          0          0          0          0        346        346          0     494550          0      12993
 12    4691764       1142          0          0          0          0        571        571          0          0          0       7316
 13    6904909       1676          0          0          0          0        838        838          0          0          0      11812
 14    3448942        796          0          0          0          0        398        398          0          0          0       7967
 15   36851438       9424          0          0         84         84       4628       4628          0          0          0       6324
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.203s ( 1269.684 blocks/s,  79.355 Mb/s)
Attachment 3: Screenshot_from_2025-02-12_08-46-38.png
Screenshot_from_2025-02-12_08-46-38.png
Attachment 4: Screenshot_from_2025-02-12_08-49-32.png
Screenshot_from_2025-02-12_08-49-32.png
Attachment 5: Screenshot_from_2025-02-12_08-50-20.png
Screenshot_from_2025-02-12_08-50-20.png
Attachment 6: Screenshot_from_2025-02-12_08-51-24.png
Screenshot_from_2025-02-12_08-51-24.png
Attachment 7: Screenshot_from_2025-02-12_08-52-57.png
Screenshot_from_2025-02-12_08-52-57.png
Attachment 8: Screenshot_from_2025-02-12_08-53-59.png
Screenshot_from_2025-02-12_08-53-59.png
Attachment 9: Screenshot_from_2025-02-12_08-55-27.png
Screenshot_from_2025-02-12_08-55-27.png
Attachment 10: Screenshot_from_2025-02-12_08-56-03.png
Screenshot_from_2025-02-12_08-56-03.png
Attachment 11: Screenshot_from_2025-02-12_08-56-46.png
Screenshot_from_2025-02-12_08-56-46.png
Attachment 12: Screenshot_from_2025-02-12_08-59-46.png
Screenshot_from_2025-02-12_08-59-46.png
Attachment 13: Screenshot_from_2025-02-12_09-01-08.png
Screenshot_from_2025-02-12_09-01-08.png
Attachment 14: Screenshot_from_2025-02-12_09-01-49.png
Screenshot_from_2025-02-12_09-01-49.png
Attachment 15: Screenshot_from_2025-02-12_09-02-30.png
Screenshot_from_2025-02-12_09-02-30.png
Attachment 16: Screenshot_from_2025-02-12_09-03-06.png
Screenshot_from_2025-02-12_09-03-06.png
Attachment 17: Screenshot_from_2025-02-12_09-04-27.png
Screenshot_from_2025-02-12_09-04-27.png
Attachment 18: Screenshot_from_2025-02-12_09-05-33.png
Screenshot_from_2025-02-12_09-05-33.png
Attachment 19: Screenshot_from_2025-02-12_09-06-04.png
Screenshot_from_2025-02-12_09-06-04.png
Attachment 20: Screenshot_from_2025-02-12_09-06-54.png
Screenshot_from_2025-02-12_09-06-54.png
Attachment 21: Screenshot_from_2025-02-12_09-07-25.png
Screenshot_from_2025-02-12_09-07-25.png
Attachment 22: Screenshot_from_2025-02-12_09-08-02.png
Screenshot_from_2025-02-12_09-08-02.png
Attachment 23: Screenshot_from_2025-02-12_09-08-47.png
Screenshot_from_2025-02-12_09-08-47.png
Attachment 24: Screenshot_from_2025-02-12_09-10-29.png
Screenshot_from_2025-02-12_09-10-29.png
Attachment 25: Screenshot_from_2025-02-12_09-12-14.png
Screenshot_from_2025-02-12_09-12-14.png
  697   Fri Jan 31 17:26:47 2025 CC, TD, MPFriday 31 January contd.
18.18 bPlas and BB7 installs complete 
      Restart AIDA DAQ, Merger, Tape Server and re-test

      per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 69 ch FWHM = 42keV FWHM

      per FEE64 Rate Spectra - attachment 2
       BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
       BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK

      per FEE64 1.8.W spectra - 20us FSR - attachments 3-5

      ADC data item stats - attachment 6
       aida01  aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
       BB7 aida10 19k

      WR timestamps OK - attachment 7

      Merger, TapeSever etc - attachments 8-9
       disk directpry /TapeData/FEB25
       working OK
     
      DSSSD bias & leakage current OK - attachment 10

      System wide checks OK - attachment 11-15
       note global clock status 6 errors reported earlier today have now gone

      FEE64 temperatures OK - attachment 16

18.38 Transition to safe state

      DAQ STOP
      disable data transfer 1
      detector bias OFF
      FEE64 power OFF


      Can restart as follows

1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
     sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps

If all OK can re-connect to Merger/TapeServer as follows

1) DAQ STOP
2) enable data transfer 1
3) DAQ GO


Can disconnect from Merger/TapeServer as follows

1) DAQ STOP
2) disable data transfer 1
   
Attachment 1: Screenshot_from_2025-01-31_18-18-17.png
Screenshot_from_2025-01-31_18-18-17.png
Attachment 2: Screenshot_from_2025-01-31_18-19-19.png
Screenshot_from_2025-01-31_18-19-19.png
Attachment 3: Screenshot_from_2025-01-31_18-19-37.png
Screenshot_from_2025-01-31_18-19-37.png
Attachment 4: Screenshot_from_2025-01-31_18-20-04.png
Screenshot_from_2025-01-31_18-20-04.png
Attachment 5: Screenshot_from_2025-01-31_18-20-30.png
Screenshot_from_2025-01-31_18-20-30.png
Attachment 6: Screenshot_from_2025-01-31_18-20-42.png
Screenshot_from_2025-01-31_18-20-42.png
Attachment 7: Screenshot_from_2025-01-31_18-21-01.png
Screenshot_from_2025-01-31_18-21-01.png
Attachment 8: Screenshot_from_2025-01-31_18-25-58.png
Screenshot_from_2025-01-31_18-25-58.png
Attachment 9: Screenshot_from_2025-01-31_18-26-13.png
Screenshot_from_2025-01-31_18-26-13.png
Attachment 10: Screenshot_from_2025-01-31_18-29-38.png
Screenshot_from_2025-01-31_18-29-38.png
Attachment 11: Screenshot_from_2025-01-31_18-29-52.png
Screenshot_from_2025-01-31_18-29-52.png
Attachment 12: Screenshot_from_2025-01-31_18-29-58.png
Screenshot_from_2025-01-31_18-29-58.png
Attachment 13: Screenshot_from_2025-01-31_18-30-06.png
Screenshot_from_2025-01-31_18-30-06.png
Attachment 14: Screenshot_from_2025-01-31_18-30-12.png
Screenshot_from_2025-01-31_18-30-12.png
Attachment 15: Screenshot_from_2025-01-31_18-30-30.png
Screenshot_from_2025-01-31_18-30-30.png
Attachment 16: Screenshot_from_2025-01-31_18-30-48.png
Screenshot_from_2025-01-31_18-30-48.png
  696   Fri Jan 31 13:48:29 2025 JBAIDA analysis from HI-DESPEC meeting 20.11.2024

AIDA analysis and presentation by J. Bormans for AIDA made in the HISPEC-DESPEC collaboration meeting.

 

https://docs.google.com/presentation/d/1hlZ30r294UqbVKGy3jg-wompFpc7TSfD/edit?usp=sharing&ouid=102131181856760114019&rtpof=true&sd=true

  695   Fri Jan 31 09:37:47 2025 CC, TD, MPFriday 31 January
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000

N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.

n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted

BB7 aida10 (asics #1-2 p+n, #3-4 n+n)

BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream) 

10.30 FEE64 power ON

      DSSSD bias ON (BB18 only) - attachment 1
       leakage current OK

      ASIC settings 2024Dec13-17.02.45 restored
       p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf

      Attachments 2-7 WR timestamp aida05 & aida12, system wide check
       aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
       WR decoder status aida02

      per FEE64 Rate spectra - attachment 8
       aida10 high rate - BB7 not biased
       rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
       BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments

      ADC data item stats - attachment 9
       rates generally OK
       aida10 high - BB7 not biased
       aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
       all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!

      BNC PB-5 settings - attachment 10

      WR timestamps OK - attachment 11

      FEE64 temperatures OK - attachment 12

12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet

      ADC data item stats - attachment 13
       all rates higher cf. attachment 9

      per FEE64 Rate spectra - attachment 14

      per p+n FEE64 1.8.L spectra - attachment 15
       aida09 pulser peak width 69 ch FWHM = 49keV FWHM 
       aida14 pulser peak width 50 ch FWHM = 35keV FWHM

      per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17

      per n+n FEE64 1.8W spectra - 20us FSR - attachment 18

      DSSSD bias & leakage current OK - attachment 19
       ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%

      FEE64 temperatures OK - attachment 20

      DSSSD bias volatge & leakage current OK - attachment 21
       CAEN N1419ET ch#0 BB18, ch#1 BB7

      Install of bPlas drivers, cabling, grounds complete

      ADC data item stats - attachment 22
       
      per FEE64 Rate spectra - attachment 23
       all BB18 p+n FEE64s *except* aida05 show very low rates of noise

      per p+n FEE64 1.8.L spectra - attachment 24
       aida09 pulser peak width 62 ch FWHM = 42keV FWHM
       BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
Attachment 1: Screenshot_from_2025-01-31_10-39-16.png
Screenshot_from_2025-01-31_10-39-16.png
Attachment 2: Screenshot_from_2025-01-31_10-58-48.png
Screenshot_from_2025-01-31_10-58-48.png
Attachment 3: Screenshot_from_2025-01-31_10-58-59.png
Screenshot_from_2025-01-31_10-58-59.png
Attachment 4: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 5: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 6: Screenshot_from_2025-01-31_10-59-52.png
Screenshot_from_2025-01-31_10-59-52.png
Attachment 7: Screenshot_from_2025-01-31_11-00-02.png
Screenshot_from_2025-01-31_11-00-02.png
Attachment 8: Screenshot_from_2025-01-31_11-17-59.png
Screenshot_from_2025-01-31_11-17-59.png
Attachment 9: Screenshot_from_2025-01-31_11-18-43.png
Screenshot_from_2025-01-31_11-18-43.png
Attachment 10: Screenshot_from_2025-01-31_11-18-50.png
Screenshot_from_2025-01-31_11-18-50.png
Attachment 11: Screenshot_from_2025-01-31_11-21-05.png
Screenshot_from_2025-01-31_11-21-05.png
Attachment 12: Screenshot_from_2025-01-31_11-21-29.png
Screenshot_from_2025-01-31_11-21-29.png
Attachment 13: Screenshot_from_2025-01-31_12-21-09.png
Screenshot_from_2025-01-31_12-21-09.png
Attachment 14: Screenshot_from_2025-01-31_12-21-55.png
Screenshot_from_2025-01-31_12-21-55.png
Attachment 15: Screenshot_from_2025-01-31_12-26-26.png
Screenshot_from_2025-01-31_12-26-26.png
Attachment 16: Screenshot_from_2025-01-31_12-30-13.png
Screenshot_from_2025-01-31_12-30-13.png
Attachment 17: Screenshot_from_2025-01-31_12-30-34.png
Screenshot_from_2025-01-31_12-30-34.png
Attachment 18: Screenshot_from_2025-01-31_12-31-25.png
Screenshot_from_2025-01-31_12-31-25.png
Attachment 19: Screenshot_from_2025-01-31_12-41-33.png
Screenshot_from_2025-01-31_12-41-33.png
Attachment 20: Screenshot_from_2025-01-31_12-42-08.png
Screenshot_from_2025-01-31_12-42-08.png
Attachment 21: Screenshot_from_2025-01-31_15-59-26.png
Screenshot_from_2025-01-31_15-59-26.png
Attachment 22: Screenshot_from_2025-01-31_16-07-21.png
Screenshot_from_2025-01-31_16-07-21.png
Attachment 23: Screenshot_from_2025-01-31_16-07-31.png
Screenshot_from_2025-01-31_16-07-31.png
Attachment 24: Screenshot_from_2025-01-31_16-08-39.png
Screenshot_from_2025-01-31_16-08-39.png
Attachment 25: Screenshot_from_2025-01-31_16-10-48.png
Screenshot_from_2025-01-31_16-10-48.png
Attachment 26: Screenshot_from_2025-01-31_16-11-30.png
Screenshot_from_2025-01-31_16-11-30.png
  694   Wed Jan 15 11:58:08 2025 TDOffline analysis data files S505 R3_9 - R3_128 (207Hg setting)
Aanlysis data files R3_9 - R3_128 (207Hg setting)

Data file R3_9 first WR ts 0x16FB110A37376D88
GMT: Wednesday, June 22, 2022 10:13:43.562 PM
Your time zone: Wednesday, June 22, 2022 11:13:43.562 PM GMT+01:00 DST

Data file R3_69 first WR ts 0x16FB2286FAD71896
GMT: Thursday, June 23, 2022 3:34:11.118 AM
Your time zone: Thursday, June 23, 2022 4:34:11.118 AM GMT+01:00 DST

Data file R3_128 first WR ts 0x16FB35DF179BD866
GMT: Thursday, June 23, 2022 9:28:40.278 AM
Your time zone: Thursday, June 23, 2022 10:28:40.278 AM GMT+01:00 DST

Attachments 1-3 - analysis data files R3_9, R_69, and R3_128
 max. *time averaged* deadtime FEE64 #1 (aida02) 15.7%, 8.6% and 15.6% respectively
 all other FEE64 deadtimes < 2%


 *** scaler( 1):            120442206  DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 2):             51126024  DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 3):              5630250  DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 4):              4983984  DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 5):             42439339  DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 6):             22316289  DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 7):                    0
 *** scaler( 8):                    0
 *** scaler( 9):                    0
 *** scaler(10):           2464294601  FEE64 #1 LEC data items
 *** scaler(11):           7641292106
 *** scaler(12):           3547670559
 *** scaler(13):           6144404472
 *** scaler(14):            637262222
 *** scaler(15):           2302716728
 *** scaler(16):           4511293677
 *** scaler(17):           3857002960  FEE64 #8 LEC data items
 *** scaler(18):                    0
 *** scaler(19):                    0
 *** scaler(20):              7697923  FEE64 #1 HEC data items
 *** scaler(21):              7169894
 *** scaler(22):             10028407
 *** scaler(23):              3045622
 *** scaler(24):              4246633
 *** scaler(25):              5564911
 *** scaler(26):              7196158
 *** scaler(27):              1960591  FEE64 #8 HEC data items
 *** scaler(28):                    0
 *** scaler(29):                    0
 *** scaler(30):                    0
 *** scaler(31):                    0
 *** scaler(32):                    0

DSSSD#1 implant events
x=m_p=0 1334587
y=m_n=0 4860983

DSSSD#2 implant events
x=m_p=0 1159736
y=m_n=0 2757735


Attachment 4 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)
 - Do not observe the off-axis energy correlations observed in the postrun background/calibration runs
 - Do observe expected inter strip charge sharing and charge sharing at DSSSD active area boundary

All subsequent spectra LEC energy difference +/-20 channels (+/- 112keV)

Attachments 5-7 - per FEE64 LEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV
Note structure in > 1500keV data not restricted to on spill

Attachments 8-10 - per FEE64 HEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Attachment 11 per DSSSD decay and implant rates (Hz) 268ms/channel =- common x and y scales
 - significant on spill deadtime for DSSSD#1

Attachment 12 - per DSSSD LEC m_p versus m_n 
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)

Attachment 13 - per DSSSD LEC p strip versus n strip

Attachment 14 - per DSSSD LEC E_p versus E_n - x and y axes 20keV/channel

Attachment 15 - per DSSSD per pixel time difference between successive LEC events (4.096us/channel)
 - minimum time difference c. 4us

Attachment 16 -  per DSSSD HEC m_p versus m_n
 - many more m=0 observed cf. RIKEN data e.g. https://elog.ph.ed.ac.uk/AIDA/583
 - more low energy HEC events?

Attachment 17 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 18 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 19 - DSSSD#1 HEC E_p versus DSSSD#2 E_p - x and y axes 20MeV/channel

Attachment 20 - DSSSD#1 LEC E_p versus DSSSD#2 E_p - x and y axes 20keV/channel
 - y=0 12816818
 - x=0 7374435
 - x>0 and y>0 2295693
 
Attachment 21 - per DSSSD per pixel time difference between successive HEC events (4.096us/channel)
 - minimum time difference c. 36us which make sense - c. readout time of ASIC with all 16 channels active
 - origin of 9 channel period spectrum structure? 

Attachments 22-23 - per DSSSD per pixel time difference between HEC and LEC events (4.096us/channel)
Attachments 24 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel)
 - for ions stopped in DSSSD#1 or DSSSD'2 - no PID selection
 - minimum time difference c. 88us
 - effect of high on spill deadtime apparent per S100 analysis

Attachment 25 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel) versus
 - HEC energy
 - LEC energy
 - p and n strip

Attachment 26 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle

Attachment 27 - per DSSSD decay event p strip - n strip time difference (2us/channel) versus decay (E_p - E_n) (20keV channel) 
Attachment 1: R3_9.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      20656
          ADC data format:  167972075 (  847983.1 Hz)
        Other data format:    1097285 (    5539.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        794 (       4.0 Hz)
                               RESUME:        793 (       4.0 Hz)
                              SYNC100:      21394 (     108.0 Hz)
                              WR48-63:      21394 (     108.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1052910 (    5315.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     290405 (    1466.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      198.084 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.300                0.000
  1               31.132                0.000
  2                0.869                0.000
  3                3.536                0.000
  4                0.001                0.000
  5                0.108                0.000
  6                0.157                0.000
  7                0.073                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13716339      17472          0          0         25         25       1820       1820          0      13782          0      46978
  1   39159612      10957          0          0        457        456       5022       5022          0          0          0      45173
  2   20693073     268290          0          0         53         53       2641       2641          0     262902          0      61677
  3   33985341     266701          0          0        182        182       4320       4320          0     257697          0      18991
  4    3620137        906          0          0          1          1        452        452          0          0          0      26130
  5   12686167       3294          0          0         28         28       1619       1619          0          0          0      34215
  6   22242523     257563          0          0         28         28       2791       2791          0     251925          0      44896
  7   21868883     272102          0          0         20         20       2729       2729          0     266604          0      12345
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   15.047s ( 1372.777 blocks/s,  85.799 Mb/s)
Attachment 2: R3_128.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260500382 (  683029.0 Hz)
        Other data format:    1419618 (    3722.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        995 (       2.6 Hz)
                               RESUME:        995 (       2.6 Hz)
                              SYNC100:      33421 (      87.6 Hz)
                              WR48-63:      33421 (      87.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1350786 (    3541.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     304165 (     797.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      381.390 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.726                0.000
  1               32.620                0.000
  2                0.268                0.000
  3                2.352                0.000
  4                0.000                0.000
  5                0.018                0.000
  6                0.191                0.000
  7                0.007                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27020397      33470          0          0        171        171       3538       3538          0      26052          0      72108
  1   70509491      19838          0          0        649        649       9270       9270          0          0          0      43322
  2   25595248     343140          0          0         48         48       3306       3306          0     336432          0      47964
  3   43087191     344986          0          0         74         74       5493       5493          0     333852          0      26057
  4    4643105       1204          0          0          0          0        602        602          0          0          0      28001
  5   18101121       4580          0          0          9          9       2281       2281          0          0          0      30316
  6   41488735     327203          0          0         39         39       5131       5131          0     316863          0      41967
  7   30055094     345197          0          0          5          5       3800       3800          0     337587          0      14430
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.207s ( 1221.046 blocks/s,  76.315 Mb/s)
Attachment 3: R3_69.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260136556 (  817109.0 Hz)
        Other data format:    1783444 (    5601.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1193 (       3.7 Hz)
                               RESUME:       1193 (       3.7 Hz)
                              SYNC100:      33186 (     104.2 Hz)
                              WR48-63:      33186 (     104.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1714686 (    5386.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     465695 (    1462.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.362 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.277                0.000
  1               49.537                0.000
  2                0.739                0.000
  3                6.304                0.000
  4                0.000                0.000
  5                0.184                0.000
  6                0.280                0.000
  7                0.034                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21719834      27795          0          0         49         49       2783       2783          0      22131          0      76464
  1   62193949      17584          0          0        707        707       8085       8085          0          0          0      68411
  2   30630417     438786          0          0         77         77       3925       3925          0     430782          0     101110
  3   52568475     429487          0          0        262        262       6724       6724          0     415515          0      29662
  4    5873809       1474          0          0          0          0        737        737          0          0          0      42098
  5   20399531       5188          0          0         27         27       2567       2567          0          0          0      56158
  6   36994221     420854          0          0         55         55       4590       4590          0     411564          0      72312
  7   29756320     442276          0          0         16         16       3775       3775          0     434694          0      19480
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.504s ( 1361.476 blocks/s,  85.092 Mb/s)
Attachment 4: Screenshot_from_2025-01-14_16-54-08.png
Screenshot_from_2025-01-14_16-54-08.png
Attachment 5: Screenshot_from_2025-01-17_08-18-33.png
Screenshot_from_2025-01-17_08-18-33.png
Attachment 6: Screenshot_from_2025-01-17_08-19-43.png
Screenshot_from_2025-01-17_08-19-43.png
Attachment 7: Screenshot_from_2025-01-17_08-20-32.png
Screenshot_from_2025-01-17_08-20-32.png
Attachment 8: Screenshot_from_2025-01-17_08-21-17.png
Screenshot_from_2025-01-17_08-21-17.png
Attachment 9: Screenshot_from_2025-01-17_08-21-58.png
Screenshot_from_2025-01-17_08-21-58.png
Attachment 10: Screenshot_from_2025-01-17_08-22-35.png
Screenshot_from_2025-01-17_08-22-35.png
Attachment 11: Screenshot_from_2025-01-17_08-23-39.png
Screenshot_from_2025-01-17_08-23-39.png
Attachment 12: Screenshot_from_2025-01-17_08-24-53.png
Screenshot_from_2025-01-17_08-24-53.png
Attachment 13: Screenshot_from_2025-01-17_08-25-25.png
Screenshot_from_2025-01-17_08-25-25.png
Attachment 14: Screenshot_from_2025-01-17_08-25-57.png
Screenshot_from_2025-01-17_08-25-57.png
Attachment 15: Screenshot_from_2025-01-17_08-27-02.png
Screenshot_from_2025-01-17_08-27-02.png
Attachment 16: Screenshot_from_2025-01-17_08-27-34.png
Screenshot_from_2025-01-17_08-27-34.png
Attachment 17: Screenshot_from_2025-01-17_08-28-03.png
Screenshot_from_2025-01-17_08-28-03.png
Attachment 18: Screenshot_from_2025-01-17_08-28-48.png
Screenshot_from_2025-01-17_08-28-48.png
Attachment 19: Screenshot_from_2025-01-17_08-29-18.png
Screenshot_from_2025-01-17_08-29-18.png
Attachment 20: Screenshot_from_2025-01-17_08-29-49.png
Screenshot_from_2025-01-17_08-29-49.png
Attachment 21: Screenshot_from_2025-01-17_08-31-27.png
Screenshot_from_2025-01-17_08-31-27.png
Attachment 22: Screenshot_from_2025-01-17_08-31-57.png
Screenshot_from_2025-01-17_08-31-57.png
Attachment 23: Screenshot_from_2025-01-17_08-32-31.png
Screenshot_from_2025-01-17_08-32-31.png
Attachment 24: Screenshot_from_2025-01-17_08-33-29.png
Screenshot_from_2025-01-17_08-33-29.png
Attachment 25: Screenshot_from_2025-01-17_08-34-12.png
Screenshot_from_2025-01-17_08-34-12.png
Attachment 26: Screenshot_from_2025-01-17_08-35-12.png
Screenshot_from_2025-01-17_08-35-12.png
Attachment 27: Screenshot_from_2025-01-17_08-36-00.png
Screenshot_from_2025-01-17_08-36-00.png
  693   Mon Jan 13 17:38:18 2025 TDOffline analysis of S505 data files
First timestamp of R2_105 0x16FAEC5E54AABF6C

GMT: Wednesday, June 22, 2022 11:01:42.903 AM
Your time zone: Wednesday, June 22, 2022 12:01:42.903 PM GMT+01:00 DST


First timestamp of R3_150 0x16FB3CE72ED53CB6

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST


First timestamp of R5_35 0x16FB89F33F9B44E2

GMT: Friday, June 24, 2022 11:09:25.825 AM
Your time zone: Friday, June 24, 2022 12:09:25.825 PM GMT+01:00 DST


First timestamp of R5_299 0x16FBD8CC5DCBE912

GMT: Saturday, June 25, 2022 11:14:20.247 AM
Your time zone: Saturday, June 25, 2022 12:14:20.247 PM GMT+01:00 DST


First timestamp of R5_546 0x16FC220EA8ADB132

GMT: Sunday, June 26, 2022 9:36:49.320 AM
Your time zone: Sunday, June 26, 2022 10:36:49.320 AM GMT+01:00 DST

Attachments 1-5 - analysis of data files R2_105, R3_150, R5_35, R5_299, and R5_536 (c. one data file per day of S505)
 FEE64 #1 (aida02) *time averaged* deadtime c. 12-18%, deadtime of all other FEE64s much lower

Attachments 6-10 - per DSSSD LEC p+n junction versus n+n Ohmic energy - X & y-axes 20keV/channel

No evidence of off-axis energy correlation observed in the postrun background/calibration runs - see https://elog.ph.ed.ac.uk/DESPEC/670
Attachment 1: R2_105.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260590543 (  842959.4 Hz)
        Other data format:    1329457 (    4300.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1451 (       4.7 Hz)
                               RESUME:       1452 (       4.7 Hz)
                              SYNC100:      33151 (     107.2 Hz)
                              WR48-63:      33151 (     107.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1260252 (    4076.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     505847 (    1636.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.138 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.956                0.000
  1               46.000                0.000
  2                0.461                0.000
  3                4.075                0.000
  4                0.270                0.000
  5               17.246                0.000
  6                0.939                0.000
  7                0.452                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27494985      28858          0          0        175        175       3688       3688          0      21132          0     102223
  1   58260423      16474          0          0        649        649       7588       7588          0          0          0      63621
  2   22639873     322607          0          0         55         55       2892       2892          0     316713          0      62482
  3   44542964     319461          0          0        153        154       5539       5539          0     308076          0      35450
  4   10779382       2700          0          0         51         51       1299       1299          0          0          0      87763
  5   28594260       7632          0          0        237        237       3579       3579          0          0          0      64742
  6   38115917     307658          0          0         84         84       4822       4822          0     297846          0      61385
  7   30162739     324067          0          0         47         47       3744       3744          0     316485          0      28181
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.383s ( 1260.696 blocks/s,  78.793 Mb/s)
Attachment 2: R3_150.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260385271 (  829605.5 Hz)
        Other data format:    1534729 (    4889.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1425 (       4.5 Hz)
                               RESUME:       1425 (       4.5 Hz)
                              SYNC100:      33170 (     105.7 Hz)
                              WR48-63:      33170 (     105.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1465539 (    4669.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     455452 (    1451.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.666                0.000
  1               57.289                0.000
  2                0.649                0.000
  3                5.611                0.000
  4                0.000                0.000
  5                0.063                0.000
  6                0.228                0.000
  7                0.039                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31441805      29767          0          0        272        272       4062       4062          0      21099          0     112151
  1   66582904      18892          0          0        814        814       8632       8632          0          0          0      66555
  2   29895404     376545          0          0         84         84       3879       3879          0     368619          0      73144
  3   44568333     369126          0          0        168        168       5484       5484          0     357822          0      38819
  4    5095090       1256          0          0          0          0        628        628          0          0          0      37415
  5   19329369       4936          0          0         23         23       2445       2445          0          0          0      43894
  6   36823085     356594          0          0         50         50       4631       4631          0     347232          0      63334
  7   26649281     377613          0          0         14         14       3409       3409          0     370767          0      20140
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.836s ( 1342.511 blocks/s,  83.907 Mb/s)
Attachment 3: R5_35.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260812842 (  842072.4 Hz)
        Other data format:    1107160 (    3574.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1210 (       3.9 Hz)
                               RESUME:       1210 (       3.9 Hz)
                              SYNC100:      33154 (     107.0 Hz)
                              WR48-63:      33154 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1038432 (    3352.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     329194 (    1062.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                8.188                0.000
  1               36.773                0.000
  2                0.176                0.000
  3                5.170                0.000
  4                0.000                0.000
  5                0.025                0.000
  6                0.202                0.000
  7                0.307                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31379366      29596          0          0        190        190       4114       4114          0      20988          0      81881
  1   59749297      16628          0          0        605        605       7709       7709          0          0          0      51815
  2   19396332     265009          0          0         42         42       2411       2411          0     260103          0      52064
  3   41019315     263381          0          0        149        149       5195       5195          0     252693          0      27341
  4    4941643       1144          0          0          0          0        572        572          0          0          0      26539
  5   17311877       4276          0          0         10         10       2128       2128          0          0          0      31104
  6   41432200     255213          0          0         55         55       5294       5294          0     244515          0      44402
  7   45582812     271913          0          0        159        159       5731       5731          0     260133          0      14048
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.660s ( 1247.070 blocks/s,  77.942 Mb/s)
Attachment 4: R5_299.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260747119 (  773887.2 Hz)
        Other data format:    1172883 (    3481.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1133 (       3.4 Hz)
                               RESUME:       1133 (       3.4 Hz)
                              SYNC100:      33256 (      98.7 Hz)
                              WR48-63:      33256 (      98.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1104105 (    3276.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     362597 (    1076.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      336.932 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                6.724                0.000
  1               44.641                0.000
  2                0.217                0.000
  3                3.561                0.000
  4                0.000                0.000
  5                0.005                0.000
  6                0.223                0.000
  7                0.026                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30674116      31250          0          0        181        181       3927       3927          0      23034          0      91358
  1   66153113      18368          0          0        695        695       8489       8489          0          0          0      55771
  2   19486494     277818          0          0         42         42       2475       2475          0     272784          0      57181
  3   43424107     279556          0          0        135        135       5543       5543          0     268200          0      30417
  4    4766466       1174          0          0          0          0        587        587          0          0          0      29048
  5   17680210       4568          0          0          5          5       2279       2279          0          0          0      34196
  6   42628036     277520          0          0         57         57       5446       5446          0     266514          0      49215
  7   35934577     282629          0          0         18         18       4510       4510          0     273573          0      15411
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.258s ( 1266.935 blocks/s,  79.183 Mb/s)
Attachment 5: R5_546.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260730517 (  745151.9 Hz)
        Other data format:    1189483 (    3399.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1144 (       3.3 Hz)
                               RESUME:       1144 (       3.3 Hz)
                              SYNC100:      33304 (      95.2 Hz)
                              WR48-63:      33304 (      95.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1120587 (    3202.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     363968 (    1040.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      349.902 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                6.803                0.000
  1               41.799                0.000
  2                0.301                0.000
  3                3.174                0.000
  4                0.000                0.000
  5                0.022                0.000
  6                0.164                0.000
  7                0.019                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29065664      31620          0          0        182        182       3682       3682          0      23892          0      90087
  1   70265931      19858          0          0        719        719       9210       9210          0          0          0      57153
  2   19423777     282104          0          0         54         54       2512       2512          0     276972          0      57467
  3   44758741     285792          0          0        125        125       5782       5782          0     273978          0      30202
  4    4551567       1158          0          0          0          0        579        579          0          0          0      29037
  5   17745398       4496          0          0         12         12       2236       2236          0          0          0      34556
  6   43164773     278410          0          0         42         42       5399       5399          0     267528          0      49766
  7   31754666     286045          0          0         10         10       3904       3904          0     278217          0      15700
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   24.879s ( 1286.230 blocks/s,  80.389 Mb/s)
Attachment 6: R2_105.png
R2_105.png
Attachment 7: R3_150.png
R3_150.png
Attachment 8: R5_35.png
R5_35.png
Attachment 9: R5_299.png
R5_299.png
Attachment 10: R5_536.png
R5_536.png
  692   Mon Jan 13 11:32:30 2025 TDOffline analysis S505 data file R3_150
Offline analysis of S505 data files R3_150

first WR ts
First timestamp of R3_150 0x16FB3CE72ED53CB6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST

last WR ts
First timestamp of R3_151 0x16FB3D3042B9E8E6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:42:45.475 AM
Your time zone: Thursday, June 23, 2022 12:42:45.475 PM GMT+01:00 DST

Analysis of data file R3_150 - attachment 1
 max time averaged deadtime FEE64 #1 (aida02) 18.3%



FEE64 configuration

FEE64   a  
      b    d
        c

         a  b  c  d 
DSSSD#1  3  4  1  2
DSSSD#2  7  8  5  6

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 100keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 *or* difference in first and last WR timestamp of event < 33us


Attachment 2 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 3 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 22 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Omic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill

Attachment 4 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 5 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 6 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- all HEC events on spill as expected
- significant deadtime on spill for n+n Ohmic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill


Attachment 7 per DSSSD decay and implant rate (Hz) 268ms/channel

Attachment 8 log scale
yellow - FEE64 aida01 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida01 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 9 log scale
yellow - FEE64 aida02 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida02 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 10 log scale
yellow - FEE64 aida05 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida05 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #2 decay rate (Hz) 268ms/channel
green  - DSSSD #2 implant rate (Hz) 268ms/channel


Attachment 11 per DSSSD LEC m_p versus m_n 

Attachment 12 per DSSSD LEC x versus y 

Attachment 13 per DSSSD LEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20keV/channel


Attachment 14 per DSSSD HEC m_p versus m_n 

Attachment 15 per DSSSD HEC x versus y 

Attachment 16 per DSSSD HEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20MeV/channel

Attachment 17 HEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20MeV/channel
 to identify which ions stop in DSSSD#2 (z_loc=2)

Attachment 18 decay & implant dx versus dy 

Attachment 19 LEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20keV/channel

Attachment 20 per DSSSD decay & implant dt (2us/channel)

Attachment 21 per DSSSD decay dt (2us/channel) versus p+n junctionj - n+n Ohmic energy difference (20keV/channel)


Summary

- high instantaneous data rates on spill
- significant deadtime FEE64 #2 on spill
   magnitude, structure, position of DSSSD#1 decay rate variations differs from FEE64 #2 deadtime - events merging?
- all other FEE64s OK on/off spill
Attachment 1: R3_150.txt
*** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260385271 (  829605.5 Hz)
        Other data format:    1534729 (    4889.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1425 (       4.5 Hz)
                               RESUME:       1425 (       4.5 Hz)
                              SYNC100:      33170 (     105.7 Hz)
                              WR48-63:      33170 (     105.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1465539 (    4669.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     455452 (    1451.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.666                0.000
  1               57.289                0.000
  2                0.649                0.000
  3                5.611                0.000
  4                0.000                0.000
  5                0.063                0.000
  6                0.228                0.000
  7                0.039                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31441805      29767          0          0        272        272       4062       4062          0      21099          0     112151
  1   66582904      18892          0          0        814        814       8632       8632          0          0          0      66555
  2   29895404     376545          0          0         84         84       3879       3879          0     368619          0      73144
  3   44568333     369126          0          0        168        168       5484       5484          0     357822          0      38819
  4    5095090       1256          0          0          0          0        628        628          0          0          0      37415
  5   19329369       4936          0          0         23         23       2445       2445          0          0          0      43894
  6   36823085     356594          0          0         50         50       4631       4631          0     347232          0      63334
  7   26649281     377613          0          0         14         14       3409       3409          0     370767          0      20140
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.836s ( 1342.511 blocks/s,  83.907 Mb/s)
Attachment 2: Screenshot_from_2025-01-12_18-48-06.png
Screenshot_from_2025-01-12_18-48-06.png
Attachment 3: Screenshot_from_2025-01-12_18-49-40.png
Screenshot_from_2025-01-12_18-49-40.png
Attachment 4: Screenshot_from_2025-01-12_18-54-53.png
Screenshot_from_2025-01-12_18-54-53.png
Attachment 5: Screenshot_from_2025-01-12_18-55-44.png
Screenshot_from_2025-01-12_18-55-44.png
Attachment 6: Screenshot_from_2025-01-12_18-56-44.png
Screenshot_from_2025-01-12_18-56-44.png
Attachment 7: Screenshot_from_2025-01-12_18-59-33.png
Screenshot_from_2025-01-12_18-59-33.png
Attachment 8: Screenshot_from_2025-01-12_19-03-44.png
Screenshot_from_2025-01-12_19-03-44.png
Attachment 9: Screenshot_from_2025-01-12_21-25-46.png
Screenshot_from_2025-01-12_21-25-46.png
Attachment 10: Screenshot_from_2025-01-12_19-08-07.png
Screenshot_from_2025-01-12_19-08-07.png
Attachment 11: Screenshot_from_2025-01-12_19-10-12.png
Screenshot_from_2025-01-12_19-10-12.png
Attachment 12: Screenshot_from_2025-01-12_19-10-51.png
Screenshot_from_2025-01-12_19-10-51.png
Attachment 13: Screenshot_from_2025-01-12_19-11-32.png
Screenshot_from_2025-01-12_19-11-32.png
Attachment 14: Screenshot_from_2025-01-12_19-12-14.png
Screenshot_from_2025-01-12_19-12-14.png
Attachment 15: Screenshot_from_2025-01-12_19-12-54.png
Screenshot_from_2025-01-12_19-12-54.png
Attachment 16: Screenshot_from_2025-01-12_19-13-22.png
Screenshot_from_2025-01-12_19-13-22.png
Attachment 17: Screenshot_from_2025-01-12_19-14-02.png
Screenshot_from_2025-01-12_19-14-02.png
Attachment 18: Screenshot_from_2025-01-12_19-14-59.png
Screenshot_from_2025-01-12_19-14-59.png
Attachment 19: Screenshot_from_2025-01-12_19-15-37.png
Screenshot_from_2025-01-12_19-15-37.png
Attachment 20: Screenshot_from_2025-01-12_19-45-39.png
Screenshot_from_2025-01-12_19-45-39.png
Attachment 21: Screenshot_from_2025-01-12_19-46-34.png
Screenshot_from_2025-01-12_19-46-34.png
Attachment 22: Screenshot_from_2025-01-13_20-04-56.png
Screenshot_from_2025-01-13_20-04-56.png
  691   Sat Jan 11 14:14:21 2025 TDaida06 crashed and rebooted 10.1.25
aida06 crashed and rebooted sometime ( 14:48 - 23:25 ) yesterday. System console log appended.


10:01:25/14:58:29|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
10:01:25/14:58:29|------------[ cut here ]------------
10:01:25/23:25:49|kernel BUG at mm/slab.c:2974!
10:01:25/23:25:49|Oops: Exception in kernel mode, sig: 5 [#1]
10:01:25/23:25:50|PREEMPT Xilinx Virtex440
10:01:25/23:25:50|Modules linked in: aidamem xdriver xh_spidev_register
10:01:25/23:25:50|NIP: c009211c LR: c00920b0 CTR: 00000006
10:01:25/23:25:50|REGS: c632dc60 TRAP: 0700   Not tainted  (2.6.31)
10:01:25/23:25:50|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
10:01:25/23:25:50|TASK = c62064c0[375] 'AidaExecV10' THREAD: c632c000
10:01:25/23:25:50|GPR00: 00000001 c632dd10 c62064c0 c680daf0 c694003c 0000000b c6940020 0000000a 
10:01:25/23:25:50|GPR08: 0000001b c680dae0 00000d80 c680dae0 22008022 10054d4c c03a0000 00000020 
10:01:25/23:25:50|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
10:01:25/23:25:50|GPR24: 00100100 c632c000 00000000 c680dae8 c680dae0 c680ae00 00000005 c680e400 
10:01:25/23:25:50|NIP [c009211c] cache_alloc_refill+0x130/0x608
10:01:25/23:25:50|LR [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:50|Call Trace:
10:01:25/23:25:50|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
10:01:25/23:25:50|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:50|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:50|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:50|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:51|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:51|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:51|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:51|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:51|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:51|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:51|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:51|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:51|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:51|Instruction dump:
10:01:25/23:25:51|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
10:01:25/23:25:51|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
10:01:25/23:25:51|Kernel panic - not syncing: Fatal exception in interrupt
10:01:25/23:25:51|Call Trace:
10:01:25/23:25:51|[c632dab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
10:01:25/23:25:51|[c632daf0] [c00345bc] panic+0x94/0x168
10:01:25/23:25:51|[c632db40] [c000bd44] die+0x178/0x18c
10:01:25/23:25:51|[c632db60] [c000c000] _exception+0x164/0x1b4
10:01:25/23:25:51|[c632dc50] [c000e978] ret_from_except_full+0x0/0x4c
10:01:25/23:25:51|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:52|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:52|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:52|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:52|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:52|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:52|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:52|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:52|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:52|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:52|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:52|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:52|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:52|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:52|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:28:52|
Finished copying zImage to RAM 
10:01:25/23:28:53|
Found 0 errors checking kernel image
10:01:25/23:28:54|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
10:01:25/23:28:54|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
10:01:25/23:28:54|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
10:01:25/23:28:54|SPIconfig done now to set up the LMK3200 registers
10:01:25/23:28:54|LMK #0 : regInit[0]=0x80000000
10:01:25/23:28:54|LMK #0 : regInit[1]=0x10070600
10:01:25/23:28:54|LMK #0 : regInit[2]=0x60601
10:01:25/23:28:54|LMK #0 : regInit[3]=0x60602
10:01:25/23:28:55|LMK #0 : regInit[4]=0x60603
10:01:25/23:28:55|LMK #0 : regInit[5]=0x70624
10:01:25/23:28:55|LMK #0 : regInit[6]=0x70605
10:01:25/23:28:55|LMK #0 : regInit[7]=0x70606
10:01:25/23:28:55|LMK #0 : regInit[8]=0x70627
10:01:25/23:28:55|LMK #0 : regInit[9]=0x10000908
10:01:25/23:28:55|LMK #0 : regInit[10]=0xA0022A09
10:01:25/23:28:55|LMK #0 : regInit[11]=0x82800B
10:01:25/23:28:55|LMK #0 : regInit[12]=0x28C800D
10:01:25/23:28:55|LMK #0 : regInit[13]=0x830020E
10:01:25/23:28:55|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
10:01:25/23:28:55|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
10:01:25/23:28:55|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
10:01:25/23:28:55|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
10:01:25/23:28:56|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
10:01:25/23:28:59|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
10:01:25/23:29:05|Using Xilinx Virtex440 machine description
10:01:25/23:29:06|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
10:01:25/23:29:06|Zone PFN ranges:
10:01:25/23:29:06|  DMA      0x00000000 -> 0x00007000
10:01:25/23:29:06|  Normal   0x00007000 -> 0x00007000
10:01:25/23:29:06|Movable zone start PFN for each node
10:01:25/23:29:06|early_node_map[1] active PFN ranges
10:01:25/23:29:06|    0: 0x00000000 -> 0x00007000
10:01:25/23:29:06|MMU: Allocated 1088 bytes of context maps for 255 contexts
10:01:25/23:29:06|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
10:01:25/23:29:06|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
10:01:25/23:29:06|PID hash table entries: 512 (order: 9, 2048 bytes)
10:01:25/23:29:07|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
10:01:25/23:29:07|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
10:01:25/23:29:07|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
10:01:25/23:29:07|Kernel virtual memory layout:
10:01:25/23:29:07|  * 0xffffe000..0xfffff000  : fixmap
10:01:25/23:29:07|  * 0xfde00000..0xfe000000  : consistent mem
10:01:25/23:29:07|  * 0xfde00000..0xfde00000  : early ioremap
10:01:25/23:29:07|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
10:01:25/23:29:07|NR_IRQS:512
10:01:25/23:29:07|clocksource: timebase mult[a00000] shift[22] registered
10:01:25/23:29:07|Console: colour dummy device 80x25
10:01:25/23:29:07|Mount-cache hash table entries: 512
10:01:25/23:29:07|NET: Registered protocol family 16
10:01:25/23:29:07|PCI: Probing PCI hardware
10:01:25/23:29:07|bio: create slab <bio-0> at 0
10:01:25/23:29:07|NET: Registered protocol family 2
10:01:25/23:29:07|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
10:01:25/23:29:07|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
10:01:25/23:29:07|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
10:01:25/23:29:07|TCP: Hash tables configured (established 4096 bind 4096)
10:01:25/23:29:08|TCP reno registered
10:01:25/23:29:08|NET: Registered protocol family 1
10:01:25/23:29:08|ROMFS MTD (C) 2007 Red Hat, Inc.
10:01:25/23:29:08|msgmni has been set to 214
10:01:25/23:29:08|io scheduler noop registered
10:01:25/23:29:08|io scheduler anticipatory registered
10:01:25/23:29:08|io scheduler deadline registered
10:01:25/23:29:08|io scheduler cfq registered (default)
10:01:25/23:29:08|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
10:01:25/23:29:08|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
10:01:25/23:29:08|console [ttyS0] enabled
10:01:25/23:29:08|brd: module loaded
10:01:25/23:29:08|loop: module loaded
10:01:25/23:29:08|Device Tree Probing 'ethernet'
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
10:01:25/23:29:08|XLlTemac: DCR address: 0x80
10:01:25/23:29:08|XLlTemac: buffer descriptor size: 32768 (0x8000)
10:01:25/23:29:08|XLlTemac: Allocating DMA descriptors with kmalloc
10:01:25/23:29:08|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
10:01:25/23:29:08|XTemac: PHY detected at address 7.
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
10:01:25/23:29:09|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09|Using buffer write method
10:01:25/23:29:09|cfi_cmdset_0001: Erase suspend on write enabled
10:01:25/23:29:09|cmdlinepart partition parsing not available
10:01:25/23:29:09|RedBoot partition parsing not available
10:01:25/23:29:09|Creating 5 MTD partitions on "fc000000.flash":
10:01:25/23:29:09|0x000000000000-0x000000500000 : "golden_firmware"
10:01:25/23:29:09|0x000000500000-0x000000800000 : "golden_kernel"
10:01:25/23:29:09|0x000000800000-0x000000d00000 : "user_firmware"
10:01:25/23:29:09|0x000000d00000-0x000000fe0000 : "user_kernel"
10:01:25/23:29:09|0x000000fe0000-0x000001000000 : "env_variables"
10:01:25/23:29:09|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32766
10:01:25/23:29:10|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32765
10:01:25/23:29:10|mice: PS/2 mouse device common for all mice
10:01:25/23:29:10|Device Tree Probing 'i2c'
10:01:25/23:29:10| #0 at 0x81600000 mapped to 0xD1030000, irq=22
10:01:25/23:29:10|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
10:01:25/23:29:10|TCP cubic registered
10:01:25/23:29:10|NET: Registered protocol family 17
10:01:25/23:29:10|RPC: Registered udp transport module.
10:01:25/23:29:10|RPC: Registered tcp transport module.
10:01:25/23:29:10|eth0: XLlTemac: Options: 0x3fa
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
10:01:25/23:29:11|eth0: XLlTemac: speed set to 1000Mb/s
10:01:25/23:29:13|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
10:01:25/23:29:13|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
10:01:25/23:29:13|Sending DHCP requests ., OK
10:01:25/23:29:14|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
10:01:25/23:29:14|IP-Config: Complete:
10:01:25/23:29:14|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
10:01:25/23:29:14|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
10:01:25/23:29:14|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
10:01:25/23:29:14|Looking up port of RPC 100003/2 on 192.168.11.99
10:01:25/23:29:14|Looking up port of RPC 100005/1 on 192.168.11.99
10:01:25/23:29:14|VFS: Mounted root (nfs filesystem) on device 0:12.
10:01:25/23:29:14|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
10:01:25/23:29:15|Starting sysinit...
10:01:25/23:29:15|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
10:01:25/23:29:16|		Press 'I' to enter interactive startup.
10:01:25/23:29:16|Setting clock  (utc): Fri Jan 10 22:29:16 GMT 2025 [  OK  ]
10:01:25/23:29:16|Building the cache [  OK  ]
10:01:25/23:29:16|Setting hostname aida06:  [  OK  ]
10:01:25/23:29:18|Mounting local filesystems:  [  OK  ]
10:01:25/23:29:19|Enabling /etc/fstab swaps:  [  OK  ]
10:01:25/23:29:21|Finishing sysinit...
INIT: Entering runlevel: 3
10:01:25/23:29:24|Entering non-interactive startup
10:01:25/23:29:24|FATAL: Module ipv6 not found.
10:01:25/23:29:26|Bringing up loopback interface:  [  OK  ]
10:01:25/23:29:29|FATAL: Module ipv6 not found.
10:01:25/23:29:29|Starting system logger: [  OK  ]
10:01:25/23:29:30|Starting kernel logger: [  OK  ]
10:01:25/23:29:30|Starting rpcbind: [  OK  ]
10:01:25/23:29:32|Mounting NFS filesystems:  [  OK  ]
10:01:25/23:29:32|Mounting other filesystems:  [  OK  ]
10:01:25/23:29:33|Starting xinetd: [  OK  ]
10:01:25/23:29:33|Starting midas:  Starting MIDAS Data Acquisition for aida06
10:01:25/23:29:34|xaida: device parameters: base=0x81000000 size=0x200000
10:01:25/23:29:39|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
10:01:25/23:29:40|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
10:01:25/23:29:40|xaida: driver assigned major number 254
10:01:25/23:29:40|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
10:01:25/23:29:45|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
10:01:25/23:29:45|AIDAMEM: aidamem: driver assigned major number 253
10:01:25/23:29:45|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
10:01:25/23:29:52|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
10:01:25/23:29:52|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
10:01:25/23:29:52|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
10:01:25/23:29:52|Computer Name = aida06; Temp Directory = /tmp/tcl361
10:01:25/23:29:57|
10:01:25/23:30:02|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
10:01:25/23:30:02|
10:01:25/23:30:02|Built without pthreads
10:01:25/23:30:02|
10:01:25/23:30:02|Creating NetVars
10:01:25/23:30:02|Output buffer length = 65504; format option = 4; transfer option = 3
10:01:25/23:30:02|EB transfer option = 3
10:01:25/23:30:02|NetVars created anxaida: open:
10:01:25/23:30:02|d initialised
10:01:25/23:30:02|SAIDAMEM: aidamem_open:
10:01:25/23:30:02|tatistics thread starting
10:01:25/23:30:02|Data Acquisition task has PID 375
10:01:25/23:30:02|Statistics thread created
10:01:25/23:30:02|Stat/Rate creation thread starting
10:01:25/23:30:02|Stat/Rate creation thread created
10:01:25/23:30:02|Hit/Rate creation thread starting
10:01:25/23:30:02|Hit/Rate creation thread created
10:01:25/23:30:02|AIDA Heartbeat thread starting
10:01:25/23:30:02|Heartbeat thread created
10:01:25/23:30:02|Installing signal handlers
10:01:25/23:30:02|Done
10:01:25/23:30:02|ModuleNum = 0
10:01:25/23:30:02|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
10:01:25/23:30:02|package limit is not available: can't find package limit
10:01:25/23:30:02|Running with default file descriptor limit
10:01:25/23:30:02|Spectra table initialised
10:01:25/23:30:02|AIDA Data Acquisition now all ready to start
10:01:25/23:30:02|SIGBUS, SIGSEGV and SIGPIPE traps setup
10:01:25/23:30:02|package setuid is not available: can't find package setuid
10:01:25/23:30:04|Running as user 0 group 0
10:01:25/23:30:04|[  OK  ]
10:01:25/23:30:12|
10:01:25/23:30:13|DENX ELDK version 4.2 build 2008-04-01
10:01:25/23:30:13|Linux 2.6.31 on a ppc
10:01:25/23:30:13|
10:01:25/23:30:13|aida06 login: /debug user "debug" password "ztl9xf9b3o0l"
10:01:25/23:30:18|httpd started on port 8015
10:01:25/23:30:18|
10:01:25/23:30:18|Cannot use /MIDAS/config/TclHttpd/aida06@8015/startup.tcl
10:01:25/23:30:18|Custom startup from /MIDAS/config/TclHttpd/aida06/startup.tcl
10:01:25/23:30:18|XAIDA Access package 1.0
10:01:25/23:30:20|/XAIDAAccessServer
10:01:25/23:30:21|XAD9252 Access package 1.0
10:01:25/23:30:21|/XAD9252AccessServer
10:01:25/23:30:21|/DataBaseAccessServer
10:01:25/23:30:21|/NetVarService
10:01:25/23:30:21|/SigTaskService
10:01:25/23:30:21|Loaded MemSasAccess
10:01:25/23:30:21|/SpectrumService
10:01:25/23:30:21|loading tcl/AIDARunControl.tcl for namespace ::
10:01:25/23:30:21|/DataAcquisitionControlServer
10:01:25/23:30:21|DefineMessage unknown
10:01:25/23:30:21|Run Control Server Implementation for AIDA
10:01:25/23:30:21|RunControlServer loaded
10:01:25/23:30:21|loading Html/RunControl/implementation.tcl
10:01:25/23:30:22|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
10:01:25/23:30:22|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
10:01:25/23:30:22|Created UI registers
10:01:25/23:30:23|RunControl loaded
10:01:25/23:30:23|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
10:01:25/23:30:23|mkdir returned 1
10:01:25/23:30:23|filepath = /MIDAS/FEE_ASIC
10:01:25/23:30:23|filename = /MIDAS/FEE_ASIC/aida06/ASICs.txt
10:01:25/23:30:23|AIDA RunControl loaded
10:01:25/23:30:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
10:01:25/23:30:24|xaida: open:
11:01:25/14:36:47|do_GetState returned z=0 and 8
11:01:25/14:59:08|do_GetState returned z=0 and 8
11:01:25/14:59:20|do_GetState returned z=0 and 8
11:01:25/14:59:32|do_GetState returned z=0 and 8
11:01:25/14:59:44|do_GetState returned z=0 and 8
11:01:25/14:59:56|do_GetState returned z=0 and 8
11:01:25/15:00:08|do_GetState returned z=0 and 8
11:01:25/15:00:20|do_GetState returned z=0 and 8
11:01:25/15:00:33|do_GetState returned z=0 and 8
11:01:25/15:00:45|do_GetState returned z=0 and 8
11:01:25/15:00:57|do_GetState returned z=0 and 8
11:01:25/15:01:09|do_GetState returned z=0 and 8
11:01:25/15:01:21|do_GetState returned z=0 and 8
11:01:25/15:01:33|do_GetState returned z=0 and 8
11:01:25/15:01:45|do_GetState returned z=0 and 8
11:01:25/15:01:58|do_GetState returned z=0 and 8
11:01:25/15:02:10|do_GetState returned z=0 and 8
11:01:25/15:02:22|do_GetState returned z=0 and 8
11:01:25/15:02:34|do_GetState returned z=0 and 8
11:01:25/15:02:46|do_GetState returned z=0 and 8
11:01:25/15:02:59|do_GetState returned z=0 and 8
11:01:25/15:03:11|do_GetState returned z=0 and 8
11:01:25/15:03:23|do_GetState returned z=0 and 8
11:01:25/15:03:35|do_GetState returned z=0 and 8
11:01:25/15:03:47|do_GetState returned z=0 and 8
11:01:25/15:03:59|do_GetState returned z=0 and 8
11:01:25/15:04:11|do_GetState returned z=0 and 8
11:01:25/15:04:24|do_GetState returned z=0 and 8
11:01:25/15:04:36|do_GetState returned z=0 and 8
11:01:25/15:04:48|do_GetState returned z=0 and 8
11:01:25/15:05:00|do_GetState returned z=0 and 8
11:01:25/15:05:12|do_GetState returned z=0 and 8
11:01:25/15:05:24|do_GetState returned z=0 and 8
11:01:25/15:05:36|do_GetState returned z=0 and 8
11:01:25/15:05:49|do_GetState returned z=0 and 8
11:01:25/15:06:01|do_GetState returned z=0 and 8
11:01:25/15:06:13|do_GetState returned z=0 and 8
11:01:25/15:06:25|do_GetState returned z=0 and 8
11:01:25/15:06:37|do_GetState returned z=0 and 8
11:01:25/15:06:50|do_GetState returned z=0 and 8
11:01:25/15:07:02|do_GetState returned z=0 and 8
11:01:25/15:07:14|do_GetState returned z=0 and 8
11:01:25/15:07:26|do_GetState returned z=0 and 8
11:01:25/15:07:38|do_GetState returned z=0 and 8
11:01:25/15:07:51|do_GetState returned z=0 and 8
11:01:25/15:08:03|do_GetState returned z=0 and 8
11:01:25/15:08:15|do_GetState returned z=0 and 8
11:01:25/15:08:27|do_GetState returned z=0 and 8
11:01:25/15:08:40|do_GetState returned z=0 and 8
11:01:25/15:08:52|do_GetState returned z=0 and 8
11:01:25/15:09:04|do_GetState returned z=0 and 8
11:01:25/15:09:17|do_GetState returned z=0 and 8
11:01:25/15:09:29|do_GetState returned z=0 and 8
11:01:25/15:09:41|do_GetState returned z=0 and 8
11:01:25/15:09:53|do_GetState returned z=0 and 8
11:01:25/15:10:06|do_GetState returned z=0 and 8
11:01:25/15:10:18|do_GetState returned z=0 and 8
11:01:25/15:10:30|do_GetState returned z=0 and 8
11:01:25/15:10:43|do_GetState returned z=0 and 8
11:01:25/15:10:55|do_GetState returned z=0 and 8
  690   Sat Jan 11 14:05:03 2025 TDSaturday 11 January 2024
14.37 Detector bias & leakage current OK - attachment 1
  
      FEE64 temperatures OK - attachment 2

      WR timestamps - attachment 3
       aida06 out of sync

      global clock status - attachment 4
       aida06 fails

      ADC calibration - attachment 5
       all fail - probably due to ASIC synchronise *after* ADC calibration

      WR decoder & FPGA TS status - attachments 6-7 
       aida06 status 0xd

      DAQ run control - attachment 8
       aida06 undefined => aida06 has rebooted

      Merger & Tape Server - attachments 9-10
       merger, tape server and data transfer have stopped

      See https://elog.ph.ed.ac.uk/DESPEC/691
       aidao06 system console - aida06 rebooted sometime yesterday


*** Recovery

For FEE64 aida06 *only*

DAQ reset
DAQ setup


For all FEE64s

DAQ stop
Restart merger, merger setup, merger go 
 Following two merger restart cycles all FEE64s stopped OK
Recovery complete

       All system wide checks OK

       Disable data transfer, DAQ go

       per FEE64 rate spectra = attachment 11
        aida04 high rates

       per FEE64 1.8.W spectra - 20us FSR - attachments 12-13

       DAQ stop, enable data transfer, DAQ go

       Merger and server consoles - attachment 14
        merger and tape server (no storage mode) running OK but no data transfer to MBS server

       WR timestamps OK - attachment 15

       ADC and DISC data item stats - attachment 16-17
        aida04 & aida16 high rates - all others appear to be OK
        ASIC check load for all aida16 ASICs changes rate from c. 420k to 0 
        ASIC check load for all aida04 ASICs changes rate from c. 200k to c. 120k

18.00 No data transfer to MBS - nothing useful being donme at the moment - will place system in safe state

      DAQ stop
      Detector bias OFF
      FEE64 power OFF
Attachment 1: Screenshot_from_2025-01-11_14-37-29.png
Screenshot_from_2025-01-11_14-37-29.png
Attachment 2: Screenshot_from_2025-01-11_14-55-21.png
Screenshot_from_2025-01-11_14-55-21.png
Attachment 3: Screenshot_from_2025-01-11_14-56-44.png
Screenshot_from_2025-01-11_14-56-44.png
Attachment 4: Screenshot_from_2025-01-11_14-57-14.png
Screenshot_from_2025-01-11_14-57-14.png
Attachment 5: Screenshot_from_2025-01-11_14-57-37.png
Screenshot_from_2025-01-11_14-57-37.png
Attachment 6: Screenshot_from_2025-01-11_14-58-05.png
Screenshot_from_2025-01-11_14-58-05.png
Attachment 7: Screenshot_from_2025-01-11_14-58-25.png
Screenshot_from_2025-01-11_14-58-25.png
Attachment 8: Screenshot_from_2025-01-11_14-59-24.png
Screenshot_from_2025-01-11_14-59-24.png
Attachment 9: Screenshot_from_2025-01-11_15-00-00.png
Screenshot_from_2025-01-11_15-00-00.png
Attachment 10: Screenshot_from_2025-01-11_15-00-31.png
Screenshot_from_2025-01-11_15-00-31.png
Attachment 11: Screenshot_from_2025-01-11_15-29-32.png
Screenshot_from_2025-01-11_15-29-32.png
Attachment 12: Screenshot_from_2025-01-11_15-30-35.png
Screenshot_from_2025-01-11_15-30-35.png
Attachment 13: Screenshot_from_2025-01-11_15-31-29.png
Screenshot_from_2025-01-11_15-31-29.png
Attachment 14: Screenshot_from_2025-01-11_15-35-14.png
Screenshot_from_2025-01-11_15-35-14.png
Attachment 15: Screenshot_from_2025-01-11_15-38-24.png
Screenshot_from_2025-01-11_15-38-24.png
Attachment 16: Screenshot_from_2025-01-11_15-27-28.png
Screenshot_from_2025-01-11_15-27-28.png
Attachment 17: Screenshot_from_2025-01-11_15-26-39.png
Screenshot_from_2025-01-11_15-26-39.png
  689   Fri Jan 10 10:03:43 2025 TDLEC fast comparator threshold too low
> > > 
> > > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > > 
> > > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > > 
> > > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> > 
> > Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> > 
> > "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> > 
> > This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending 
data. (@Nic is this correct?)
> > 
> > This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."
> 
> We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.
> 
> We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.

Per our discussions in December 2024 this is exactly what would be expected.
  688   Thu Jan 9 17:56:14 2025 TDLEC fast comparator threshold too low
> > 
> > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > 
> > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > 
> > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> 
> Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> 
> "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> 
> This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)
> 
> This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."

We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.

We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.
  687   Thu Jan 9 17:36:50 2025 TDLEC fast comparator threshold too low
> 
> LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> 
> The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> 
> I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.

Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18

"We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.

This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)

This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."
  686   Thu Jan 9 16:25:51 2025 TDLEC fast comparator threshold too low
LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.

The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).

I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
Attachment 1: Screenshot_from_2025-01-09_17-28-45.png
Screenshot_from_2025-01-09_17-28-45.png
  685   Thu Jan 9 15:43:17 2025 JB, CC, MPTiming test platform moved in

16:30 We moved the platform in. The system was still biased and the DAQ was still running.

The system seems to be okay, temperatures OK. Bias voltage OK 685/1 and 685/2. Two FEEs - aida07 and aida12 are receiving a lot of trigger data items from the fast discriminator. This is already a bit worrying. 685/3

Attachment 1: Screenshot_from_2025-01-09_16-41-48.png
Screenshot_from_2025-01-09_16-41-48.png
Attachment 2: Screenshot_from_2025-01-09_16-42-06.png
Screenshot_from_2025-01-09_16-42-06.png
Attachment 3: Screenshot_from_2025-01-09_16-42-46.png
Screenshot_from_2025-01-09_16-42-46.png
  684   Thu Jan 9 12:46:43 2025 JB, MP, CCAIDA timing test

 

Quote:

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

 

Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03
Attachment 1: AIDAfasttime_bPlastWRdt_AIDA10.png
AIDAfasttime_bPlastWRdt_AIDA10.png
Attachment 2: AIDACalAdcFasttime.png
AIDACalAdcFasttime.png
Attachment 3: AIDAfasttime_bPlastWRdt_AIDA10_nopulser.png
AIDAfasttime_bPlastWRdt_AIDA10_nopulser.png
Attachment 4: Centrepeak_gauss_fit.png
Centrepeak_gauss_fit.png
  683   Wed Jan 8 16:47:05 2025 JB, MP, CCAIDA timing test

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

Attachment 1: Screenshot_from_2025-01-08_17-51-51.png
Screenshot_from_2025-01-08_17-51-51.png
  682   Wed Jan 8 10:46:50 2025 TDReport - aida06 -boot fail due to network issue
8:01:25/11:42:48|0x000000d00000-0x000000fe0000 : "user_kernel"
08:01:25/11:42:48|0x000000fe0000-0x000001000000 : "env_variables"
08:01:25/11:42:48|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32766
08:01:25/11:42:48|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32765
08:01:25/11:42:48|mice: PS/2 mouse device common for all mice
08:01:25/11:42:48|Device Tree Probing 'i2c'
08:01:25/11:42:48| #0 at 0x81600000 mapped to 0xD1030000, irq=22
08:01:25/11:42:48|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
08:01:25/11:42:48|TCP cubic registered
08:01:25/11:42:48|NET: Registered protocol family 17
08:01:25/11:42:48|RPC: Registered udp transport module.
08:01:25/11:42:49|RPC: Registered tcp transport module.
08:01:25/11:42:49|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:42:49|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:42:51|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:42:51|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:42:51|Sending DHCP requests .
08:01:25/11:42:53|eth0: XLlTemac: PHY Link carrier lost.
08:01:25/11:42:53|..... timed out!
08:01:25/11:44:15|IP-Config: Reopening network devices...
08:01:25/11:44:15|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:44:16|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:44:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:44:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:44:18|Sending DHCP requests ...... timed out!
08:01:25/11:45:35|IP-Config: Auto-configuration of network failed.
08:01:25/11:45:35|Root-NFS: No NFS server available, giving up.
08:01:25/11:45:35|VFS: Unable to mount root fs via NFS, trying floppy.
08:01:25/11:45:35|VFS: Cannot open root device "nfs" or unknown-block(2,0)
08:01:25/11:45:35|Please append a correct "root=" boot option; here are the available partitions:
08:01:25/11:45:35|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
08:01:25/11:45:35|Call Trace:
08:01:25/11:45:35|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
08:01:25/11:45:35|[c6827f10] [c00345bc] panic+0x94/0x168
08:01:25/11:45:35|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
08:01:25/11:45:36|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
08:01:25/11:45:36|[c6827fd0] [c0341220] kernel_init+0x104/0x130
08:01:25/11:45:36|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
08:01:25/11:45:36|Rebooting in 180 seconds..
  681   Wed Jan 8 10:08:29 2025 JB, GB, SD, MP, CC, JGAIDA noise test with platform in position

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

Attachment 1: Screenshot_from_2025-01-09_09-32-08.png
Screenshot_from_2025-01-09_09-32-08.png
Attachment 2: Screenshot_from_2025-01-09_09-35-11.png
Screenshot_from_2025-01-09_09-35-11.png
Attachment 3: Screenshot_from_2025-01-09_09-32-20.png
Screenshot_from_2025-01-09_09-32-20.png
Attachment 4: Screenshot_from_2025-01-09_09-31-58.png
Screenshot_from_2025-01-09_09-31-58.png
  680   Mon Dec 30 15:43:42 2024 TD[How To] Restart AnyDesk
See also https://elog.ph.ed.ac.uk/CARME/489

1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)
  679   Mon Dec 23 10:54:46 2024 TDMonday 23 December 2024
11.54 re-established remote access via AnyDesk - details to follow

      confirmed AIDA FEE64 power OFF, detector HV OFF


1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)
  678   Tue Dec 17 12:45:36 2024 JB, MP, CCAIDA timing test

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV)

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

678/1 shows the AIDA OR64 trigger from the fast comparator of aida10.

678/2-4 show the bPlast accepted trigger after the bPlast DAQ has been triggered by the AIDA OR64. The images show a signal coming fast in time after the pulser and also a signal at around 80 us that appears to be coming from pile up.

The bPlast DAQ also now runs with the AIDA OR64 trigger after the NIM out signal from the MACB was sent to an octal discriminator to fix the width of the signal and also the pulse width which from TAMEX should be kept at or above 100 ns. The NIM signal from the MACB was mostly 100 ns but also jumping to signals with a 20ns pulse width.

 

Attachment 1: IMG_2536.jpeg
IMG_2536.jpeg
Attachment 2: 20241217_141148.jpg
20241217_141148.jpg
Attachment 3: 20241217_141206.jpg
20241217_141206.jpg
  677   Mon Dec 16 13:07:04 2024 JB, MP, CCAIDA timing test

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

Attachment 1: Screenshot_from_2024-12-17_10-15-42.png
Screenshot_from_2024-12-17_10-15-42.png
Attachment 2: Screenshot_from_2024-12-17_10-27-00.png
Screenshot_from_2024-12-17_10-27-00.png
Attachment 3: Screenshot_from_2024-12-17_10-21-15.png
Screenshot_from_2024-12-17_10-21-15.png
Attachment 4: Screenshot_from_2024-12-17_10-21-55.png
Screenshot_from_2024-12-17_10-21-55.png
Attachment 5: Screenshot_from_2024-12-17_10-22-20.png
Screenshot_from_2024-12-17_10-22-20.png
Attachment 6: Screenshot_from_2024-12-17_10-23-27.png
Screenshot_from_2024-12-17_10-23-27.png
Attachment 7: Screenshot_from_2024-12-17_10-23-32.png
Screenshot_from_2024-12-17_10-23-32.png
Attachment 8: Screenshot_from_2024-12-17_10-23-37.png
Screenshot_from_2024-12-17_10-23-37.png
Attachment 9: Screenshot_from_2024-12-17_10-23-42.png
Screenshot_from_2024-12-17_10-23-42.png
Attachment 10: Screenshot_from_2024-12-17_10-35-07.png
Screenshot_from_2024-12-17_10-35-07.png
Attachment 11: Screenshot_from_2024-12-17_10-39-36.png
Screenshot_from_2024-12-17_10-39-36.png
Attachment 12: Screenshot_from_2024-12-17_13-35-40.png
Screenshot_from_2024-12-17_13-35-40.png
Attachment 13: Screenshot_from_2024-12-17_13-35-45.png
Screenshot_from_2024-12-17_13-35-45.png
Attachment 14: Screenshot_from_2024-12-17_13-35-51.png
Screenshot_from_2024-12-17_13-35-51.png
Attachment 15: Screenshot_from_2024-12-17_13-35-56.png
Screenshot_from_2024-12-17_13-35-56.png
Attachment 16: Screenshot_from_2024-12-17_13-36-01.png
Screenshot_from_2024-12-17_13-36-01.png
  676   Sat Dec 14 11:52:42 2024 TDSaturday 14 December
11:00 Visual inspection FEE64 adaptor PCBs & cabling
      aida02 ground from/to other FEE64s disconnected - re-connected
      aida01, aida09, aida12 - ground cabling screws to Lemo 00.250 housings loose - tightemed

12:50 Cooling water pressure & temperature OK - attachments 1  2

12:59 relay #1 power ON

13:01 relay #2 power ON

13:07 aida06 starts - panic during startup, automatic restart following 3 minute timeout

      DAQ reset, setup

      Check ASIC Control - browser tab timeout 
      AIDA MIDAS HTTPD server console log - attachment 3

      Appears to have restored ASIC settings 2024Dec13-17.02.45 saved yesterday
    
      aida10 ASICs #1 & #2 positive input, ASICs #3 & #4 negative input
      slow comparator 0xa (all p+n junction FEE64s and aida10), 0xf (n+n Ohmic FEE64s)

13:28 tar ASIC settings - attachment 4

[npg@aidas-gsi]$ cd /MIDAS/FEE_ASIC
[npg@aidas-gsi FEE_ASIC]$ tar cvf /tmp/FEE_ASIC.tar .

       System wide checks

       Sync ASIC clocks - attachment 5

       Clock, ADC calibration, WR decoder, FPGA timestamp, PLL checks - attachments 6-11
        all OK *except* aida02 WR decoder error

       WR timestamps OK - attachment 12
       FEE64 temps OK - attachment 13
       
13:45 Detector bias ON - attachment 14      

      BNC PB-5 pulser settings - attachment 15
      Pulser connected to all p+n junction FEE64s *except* aida10

      ADC, DISC, PAUSE and MBS correlation scaler stats - attachments 16-19
       aida02 rate significantly lower than yesterday - https://elog.gsi.de/despec/Implantation+Stack/8
       high rates observed for aida08, aida11 and aida14 - which are not connected to a DSSSD!

      per FEE64 Rate spectra - attachment 20

      per p+n junction FEE64 1.8.L spectra - attachment 21
       aida09 pulser peak width 56 ch FWHM = 39 keV FWHM
       consistent electronic noise for all p+n junction FEE64s (cabling+DSSSD)
       electronic noise of p+n junction FEE64s (cabling *only*) higher and more variable cf. https://elog.gsi.de/despec/Implantation+Stack/8 attachment 5

      per p+n junction FEE64 1.8.W spectra 20us FSR - attachments 22-23

      per n+n Ohmic FEE64 1.8.W spectra 20us FSR - attachment 24 

      WR timestamps OK - attachment 25

14:43 DAQ STOP
      Data transfer enabled
      Select Tape Server -> Next Run
      DAQ GO  data file R3

      Merger, Tape Server - attachments 26-27
       data transfer rate c. 900k data items/s cf c. 300k data items/s yesterday https://elog.gsi.de/despec/Implantation+Stack/8

16:45 DAQ STOP

      Data transfer disabled 

      Detector bias OFF

      FEE64 power OFF
Attachment 1: 20241214_125048.jpg
20241214_125048.jpg
Attachment 2: 20241214_125104.jpg
20241214_125104.jpg
Attachment 3: asic_check.log
executing do_PreReset
 ::MASTERTS namespace already exists
done executing do_PreReset
executing do_PostSetup
done executing do_PostSetup 
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida01
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida01
Second load re-activated at 0x0000401c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida01
Second load re-activated at 0x0000405c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida01
Second load re-activated at 0x0000409c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida01
Second load re-activated at 0x000040dc  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida02
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida02
Second load re-activated at 0x0000401c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida02
Second load re-activated at 0x0000405c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida02
Second load re-activated at 0x0000409c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida02
Second load re-activated at 0x000040dc  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida03
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida03
Second load re-activated at 0x0000401c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida03
Second load re-activated at 0x0000405c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida03
Second load re-activated at 0x0000409c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida03
Second load re-activated at 0x000040dc  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida04
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida04
Second load re-activated at 0x0000401c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida04
Second load re-activated at 0x0000405c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida04
Second load re-activated at 0x0000409c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida04
Second load re-activated at 0x000040dc  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida05
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida05
Second load re-activated at 0x0000401c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida05
Second load re-activated at 0x0000405c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida05
Second load re-activated at 0x0000409c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida05
Second load re-activated at 0x000040dc  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida06
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida06
Second load re-activated at 0x0000401c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida06
Second load re-activated at 0x0000405c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida06
Second load re-activated at 0x0000409c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida06
Second load re-activated at 0x000040dc  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida07
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida07
Second load re-activated at 0x0000401c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida07
Second load re-activated at 0x0000405c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida07
Second load re-activated at 0x0000409c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida07
Second load re-activated at 0x000040dc  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida08
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida08
Second load re-activated at 0x0000401c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida08
Second load re-activated at 0x0000405c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida08
Second load re-activated at 0x0000409c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida08
Second load re-activated at 0x000040dc  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida09
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida09
Second load re-activated at 0x0000401c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida09
Second load re-activated at 0x0000405c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida09
Second load re-activated at 0x0000409c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida09
Second load re-activated at 0x000040dc  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida10
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida10
Second load re-activated at 0x0000401c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida10
Second load re-activated at 0x0000405c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida10
Second load re-activated at 0x0000409c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida10
Second load re-activated at 0x000040dc  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida11
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida11
Second load re-activated at 0x0000401c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida11
Second load re-activated at 0x0000405c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida11
Second load re-activated at 0x0000409c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida11
Second load re-activated at 0x000040dc  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida12
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida12
Second load re-activated at 0x0000401c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida12
Second load re-activated at 0x0000405c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida12
Second load re-activated at 0x0000409c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida12
Second load re-activated at 0x000040dc  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida13
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida13
Second load re-activated at 0x0000401c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida13
Second load re-activated at 0x0000405c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida13
Second load re-activated at 0x0000409c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida13
Second load re-activated at 0x000040dc  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida14
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida14
Second load re-activated at 0x0000401c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida14
Second load re-activated at 0x0000405c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida14
Second load re-activated at 0x0000409c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida14
Second load re-activated at 0x000040dc  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida15
... 379 more lines ...
Attachment 4: FEE_ASIC.tar
Attachment 5: Screenshot_from_2024-12-14_13-30-48.png
Screenshot_from_2024-12-14_13-30-48.png
Attachment 6: Screenshot_from_2024-12-14_13-30-54.png
Screenshot_from_2024-12-14_13-30-54.png
Attachment 7: Screenshot_from_2024-12-14_13-31-02.png
Screenshot_from_2024-12-14_13-31-02.png
Attachment 8: Screenshot_from_2024-12-14_13-35-22.png
Screenshot_from_2024-12-14_13-35-22.png
Attachment 9: Screenshot_from_2024-12-14_13-35-28.png
Screenshot_from_2024-12-14_13-35-28.png
Attachment 10: Screenshot_from_2024-12-14_13-35-33.png
Screenshot_from_2024-12-14_13-35-33.png
Attachment 11: Screenshot_from_2024-12-14_13-36-06.png
Screenshot_from_2024-12-14_13-36-06.png
Attachment 12: Screenshot_from_2024-12-14_13-36-27.png
Screenshot_from_2024-12-14_13-36-27.png
Attachment 13: Screenshot_from_2024-12-14_13-36-42.png
Screenshot_from_2024-12-14_13-36-42.png
Attachment 14: Screenshot_from_2024-12-14_13-44-29.png
Screenshot_from_2024-12-14_13-44-29.png
Attachment 15: Screenshot_from_2024-12-14_13-46-41.png
Screenshot_from_2024-12-14_13-46-41.png
Attachment 16: Screenshot_from_2024-12-14_13-54-58.png
Screenshot_from_2024-12-14_13-54-58.png
Attachment 17: Screenshot_from_2024-12-14_13-55-19.png
Screenshot_from_2024-12-14_13-55-19.png
Attachment 18: Screenshot_from_2024-12-14_13-55-33.png
Screenshot_from_2024-12-14_13-55-33.png
Attachment 19: Screenshot_from_2024-12-14_13-56-00.png
Screenshot_from_2024-12-14_13-56-00.png
Attachment 20: Screenshot_from_2024-12-14_14-10-08.png
Screenshot_from_2024-12-14_14-10-08.png
Attachment 21: Screenshot_from_2024-12-14_14-14-18.png
Screenshot_from_2024-12-14_14-14-18.png
Attachment 22: Screenshot_from_2024-12-14_14-26-30.png
Screenshot_from_2024-12-14_14-26-30.png
Attachment 23: Screenshot_from_2024-12-14_14-27-16.png
Screenshot_from_2024-12-14_14-27-16.png
Attachment 24: Screenshot_from_2024-12-14_14-29-24.png
Screenshot_from_2024-12-14_14-29-24.png
Attachment 25: Screenshot_from_2024-12-14_14-42-40.png
Screenshot_from_2024-12-14_14-42-40.png
Attachment 26: Screenshot_from_2024-12-14_14-43-51.png
Screenshot_from_2024-12-14_14-43-51.png
Attachment 27: Screenshot_from_2024-12-14_14-43-58.png
Screenshot_from_2024-12-14_14-43-58.png
  675   Wed Dec 11 14:46:21 2024 TD, JB, CC, MPMounting of AIDA, bPlast and BB7 for test

Snout assembled for detector position and timing test.

The planned test is to use a BGO with a 22-Na source and the implantation stack to characterise the timing and position of back-to-back 511 keV events (proxy for mock beta decay event).

The implantation stack is assembled as follows:

Upstream ----> AIDA test frame (as a spacer) ----> AIDA DSSSD (3208-3/3208-21/3208-22, Upstream AIDA used in S100 & S181, April 2024) ----> bPlast (Downstream bPlast) ----> BB7 (mounted in test frame connected to AIDA adaptor) ----> Downstream

The AIDA DSSSD condition seems to be the same as when it was dismounted after S181 and it, apart from some specks of dust no bond wires appear to be broken to the naked eye.

BGO signals appear on the scope with a 22-Na source but the response is quite messy.

 

 

17:36

MSL type BB18(DS)-1000 24cm  x 8cm DSSSD upstream position within snout using 8x upstream (23") ribbon cables 

MSL type BB(DS)-1000 positioned downstream of downstream bPlast within snout using 1x downstream (29") ribbon cable. For BB7 'isolated' Right Coupler Kapton PCB replaced by unmodified Right Coupler Kapton PCB.

 

 

 

Attachment 1: 20241211_161837.jpg
20241211_161837.jpg
Attachment 2: 20241211_161833.jpg
20241211_161833.jpg
Attachment 3: 20241211_161830.jpg
20241211_161830.jpg
Attachment 4: 20241211_161820.jpg
20241211_161820.jpg
  674   Mon Dec 9 12:28:01 2024 NH, CC, MPTest of BB7 on AIDA Electronics
BB7 connected to aida07, sitting in a cardboard box

Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015

Power on all FEEs

Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
 No, but a powercycle worked and fixed it

See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt  not found, create the folders

We see some data in the L spectra, and changing the settings from one ASIC seems to work properly

Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n

Follow up: Alpha data overnight or source?
  673   Thu Dec 5 11:18:46 2024 TDOffline analysis S100 data files R21_0-R21_99
Offline analysis of S100 data files R21_0 - R21_99 (162Eu setting)

first WR ts
First timestamp of R21_0 0x17CA09154AE3E636

Epoch converter says ...

GMT: Saturday, April 27, 2024 4:36:35.223 AM
Your time zone: Saturday, April 27, 2024 5:36:35.223 AM GMT+01:00 DST

last WR ts
First timestamp of R21_100 0x17CA16C1904150CE

GMT: Saturday, April 27, 2024 8:47:08.772 AM
Your time zone: Saturday, April 27, 2024 9:47:08.772 AM GMT+01:00 DST


FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2 
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500

Attachments 1-4
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- observe large (> 32us) time differences (on log scale)

- range of time differences increases with multiplicity ( DSSSD#1 cf. DSSSD#2 LEC events)

- distribution of HEC time differences can probably be understood in terms of most/all channels of ASIC being active during HEC event with low LEC thresholds

- AIDA is a triggerless DAQ producing streams of ADC data items *not* events
   at high instantaneous rates when events are constructed they may become aggregated in time i.e. > 32us readout time of all channels of one ASIC

- To investigate impose additional end of event criterion 
   difference in first and last WR timestamp of event < 33us

Attachments 5-6
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- blue original end of event criteria, cyan new end of event criteria

- as expected range of time differences is restricted to +/- 32us 

- observe somewhat higher fraction of events with low time differences
 
  DSSSD #1 10363098 of 16104322 (64%) events +/-2us 

  DSSSD #2 860454912 of 1766618199 (49%) events +/-2us

Attachment 7
per DSSSD p+n junction - n+n Ohmic strip time difference for LEC events - x-axis 2us/channel, y-axis 20keV /channel


Attachment 8 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 9 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 10 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 11 per FEE64 LEC hit pattern: 150keV < energy < 1500keV
Attachment 12 per FEE64 LEC hit pattern: energy > 1500keV


Attachment 13 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 14 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 15 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- rate dominated by low energy (>1GeV) events
- all HEC events on spill as expected (note FEE64 #7 has a single hot channel which can be disabled in software)
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 16 per DSSSD p+n junction versus n+n Ohmic LEC energy - x-axis & y-axis 20keV/channel

Attachment 17 per DSSSD p+n junction versus n+n Ohmic HEC energy - x-axis & y-axis 20MeV/channel

Attachment 18 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern: all HEC events

Attachment 19 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern
 DSSSD #1 ions stopped in DSSSD #1 i.e. DSSSD #2 HEC multiplicity = 0
 DSSSD #1 shows x-y gate used ( 270 < x < 370, 20 < y < 90 ) to identify 166Tb implants
 DSSSD #2 ions stopped in DSSSD #2 *and* in transmission (can establish which ions stop in DSSSD#2 from DSSSD#2 HEC energy versus DSSSD#1 HEC energy - see https://elog.ph.ed.ac.uk/DESPEC/672

Attachment 20

DSSSD#1 HEC energy (20MeV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 LEC energy (20keV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 HEC strip # versus HEC-LEC dt (1s/channel)

- Observe # events in every third channel is lower 
- Probably reflects implant-decay correlation livetime
  For example (choosing some numbers for illustrative purposes)
  on spill: HEC livetime 75%, LEC livetime 75% (FEE64 deadtime common for HEC and LEC data) => implant decay correlation livetime 56%
  off spill: HEC live time 75%, LEC livetime 100% =>  implant decay correlation livetime 75%
- Observe 'hot' x channels 315, 318, 321, 324 - disabled for further analysis
- Do not observe any 'hot' y channels 


Attachments 21 & 22

DSSSD#1 per pixel HEC-LEC time (1s/channel): x,y,z gated to select 166Tb events

Naive (parent-daughter decay only, flat background) fit for data t=0-26s ( t1/2 = 27.1(3)s )

Fit ignores data for t=0, 3, 6, 10, 13, 16, 19, 22s to avoid bias of differences in implant-decay correlation deadtime

Suggestion of structure at c. 30s period? Does this reflect spill stucture? 10x spill cycles (30s), 9s spill off, ... etc

Sum of x,y,z gated HEC events (s2112 - see attachment 19) = 670441

Elapsed time of dataset 4h11m = 15060s
# pixels = 100 x 70 = 7000
=> # x,y,z gated HEC events per pixel = 670441/7000/15060 = 0.0064/s or mean time between x,y,z gated HEC event = 157s (estimate needs to be corrected for HEC deadtime)

Sum of implant decay correlations (s2220 - see attachment 21) t=0-150s = 273508 - flat background estimated as 150 x 500 = 75000 = 198508

=> efficiency c. 30% (presumably low due to implant-decay deadtime, LEC multiplicity, per pixel correlations and no clustering)


Summary

$64,000 question - what is the origin of the high instantaneous rate on spill  for DSSSD#1 ? On my to do list.


Attachment 23

LEC multiplicity with/without HEC data in event

per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity
per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity z_hec=1 and z_hec=2


With HEC data

DSSSD#1 p+n junction multiplicity ~ 17, n+n Ohmic multiplicity ~28
DSSSD#2 p+n junction multiplicity ~ 40, n+n Ohmic multiplicity ~23

Assume 200Hz HEC events => DSSSD#1 LEC rate = 200 x ( 17 + 28 ) => 9k LEC data items cf. >100k LEC data items (attachments 8 & 13)

i.e. not due to HEC events








 
 


 







   
Attachment 1: Screenshot_from_2024-12-05_11-17-50.png
Screenshot_from_2024-12-05_11-17-50.png
Attachment 2: Screenshot_from_2024-12-05_11-18-13.png
Screenshot_from_2024-12-05_11-18-13.png
Attachment 3: Screenshot_from_2024-12-05_11-04-40.png
Screenshot_from_2024-12-05_11-04-40.png
Attachment 4: Screenshot_from_2024-12-05_11-05-03.png
Screenshot_from_2024-12-05_11-05-03.png
Attachment 5: Screenshot_from_2024-12-05_11-07-58.png
Screenshot_from_2024-12-05_11-07-58.png
Attachment 6: Screenshot_from_2024-12-05_11-09-55.png
Screenshot_from_2024-12-05_11-09-55.png
Attachment 7: Screenshot_from_2024-12-04_16-56-02.png
Screenshot_from_2024-12-04_16-56-02.png
Attachment 8: Screenshot_from_2024-12-04_16-39-09.png
Screenshot_from_2024-12-04_16-39-09.png
Attachment 9: Screenshot_from_2024-12-04_16-39-59.png
Screenshot_from_2024-12-04_16-39-59.png
Attachment 10: Screenshot_from_2024-12-04_16-41-20.png
Screenshot_from_2024-12-04_16-41-20.png
Attachment 11: Screenshot_from_2024-12-04_16-41-54.png
Screenshot_from_2024-12-04_16-41-54.png
Attachment 12: Screenshot_from_2024-12-04_16-42-37.png
Screenshot_from_2024-12-04_16-42-37.png
Attachment 13: Screenshot_from_2024-12-04_16-43-36.png
Screenshot_from_2024-12-04_16-43-36.png
Attachment 14: Screenshot_from_2024-12-04_16-44-13.png
Screenshot_from_2024-12-04_16-44-13.png
Attachment 15: Screenshot_from_2024-12-04_16-44-50.png
Screenshot_from_2024-12-04_16-44-50.png
Attachment 16: Screenshot_from_2024-12-04_16-46-26.png
Screenshot_from_2024-12-04_16-46-26.png
Attachment 17: Screenshot_from_2024-12-04_16-50-32.png
Screenshot_from_2024-12-04_16-50-32.png
Attachment 18: Screenshot_from_2024-12-04_16-49-12.png
Screenshot_from_2024-12-04_16-49-12.png
Attachment 19: Screenshot_from_2024-12-04_16-49-57.png
Screenshot_from_2024-12-04_16-49-57.png
Attachment 20: Screenshot_from_2024-12-04_16-52-09.png
Screenshot_from_2024-12-04_16-52-09.png
Attachment 21: Screenshot_from_2024-12-04_16-51-30.png
Screenshot_from_2024-12-04_16-51-30.png
Attachment 22: 166Tb.png
166Tb.png
Attachment 23: Screenshot_from_2024-12-07_16-57-28.png
Screenshot_from_2024-12-07_16-57-28.png
  672   Wed Dec 4 09:57:24 2024 TDHISPEC DESPEC meeting presentation - November 2024 -0 Summary of AIDA performance 2024
Attachment 1: Summary_of_AIDA_2024_distro.pdf
Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf
  671   Tue Nov 19 11:24:25 2024 TDAnydesk restarted remotely
Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489
  670   Tue Nov 5 15:19:35 2024 TDS505 offline analysis R5_780 - R5_814
Offline analysis of data files S505/R5_780 - R5_814 (corresponding to MBS data files 73-74) 

Can find an analysis of alpha background runs at RIBF, RIKEN for comparison at https://elog.ph.ed.ac.uk/AIDA/816

Beam off - background runs without sources

MBS 73 27.6.22 09:33-11:33 CEST https://elog.gsi.de/despec/S505/204

MBS 74 27.6.22 11:34-13:34 CEST https://elog.gsi.de/despec/S505/206

MBS 75 27.6.22 13:35-16:29 CEST https://elog.gsi.de/despec/S505/209

ADC offsets per https://elog.ph.ed.ac.uk/DESPEC/556

FEE64 configuration

FEE64   a b c d
DSSSD#1 3 4 1 2
DSSSD#2 7 8 5 6

p+n junction FE64s odd numbered


Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
- no clustering
- no p+n junction side - n+n Ohmic side correlation time gates
- valid LEC events
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 


Attachments 1-2 - per DSSSD p+n versus n+n multiplicity

Attachment 3 - per DSSSD x versus y

Attachments 4-5 - per DSSSD p+n versus n+n energy (20keV/channel nominal)
                  all combinations of per DSSSD p+n junction and n+n Ohmic energies 
                  with projection of data within window onto x and y axes

                  too many events for natural (U decay series) background
                  off leading diagonal correlations anomalous
                  transverse width of leading diagonal correlation wider than expected - ADC offsets OK?
                 

Attachment 6 - per FEE64 WR timestamp (32.768us/channel)
               FEE64 sync test using pulser data - looks OK

Attachments 7-14 - per FEE64 ADC spectra (5.6keV/channel nominal)
                   note common x/y scale - pulser peak height proxy for peak width

per FEE64 1.8.L pulser peak widths (ch FWHM)

1 11.43
2 16.58
3 12.52
4 17.54
5 9.10
6 12.22
7 17.03
8 16.09

3 of 4 p+n junction FEE64 good (<70keV), 1 of 4 n+n Ohmic FEE64 good - all others < 100keV
Pulser peak indicates noise/gain/offset stable throughout background runs


Attachments 15-22 - per FEE64 ADC spectra (5.6kleV/channel nominal)
                    observe broad peak-like structures (c. 2.8MeV) in first channel of most ASICs? 

Attachment 23 per DSSSD p+n versus n+n energy (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachment 24 p+n FEE64 #0 energy versus each of n+n FEE64s (#1, #3, #5 and #7) energies (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachments 23 & 24 do not unambiguously identify which FEE64s are attached to which DSSSD
                  
Attachment 1: Screenshot_from_2024-11-06_10-16-55.png
Screenshot_from_2024-11-06_10-16-55.png
Attachment 2: Screenshot_from_2024-11-06_10-18-43.png
Screenshot_from_2024-11-06_10-18-43.png
Attachment 3: Screenshot_from_2024-11-06_10-19-38.png
Screenshot_from_2024-11-06_10-19-38.png
Attachment 4: Screenshot_from_2024-11-06_10-21-13.png
Screenshot_from_2024-11-06_10-21-13.png
Attachment 5: Screenshot_from_2024-11-06_10-23-42.png
Screenshot_from_2024-11-06_10-23-42.png
Attachment 6: Screenshot_from_2024-11-06_10-47-04.png
Screenshot_from_2024-11-06_10-47-04.png
Attachment 7: Screenshot_from_2024-11-06_10-49-16.png
Screenshot_from_2024-11-06_10-49-16.png
Attachment 8: Screenshot_from_2024-11-06_10-50-17.png
Screenshot_from_2024-11-06_10-50-17.png
Attachment 9: Screenshot_from_2024-11-06_10-50-59.png
Screenshot_from_2024-11-06_10-50-59.png
Attachment 10: Screenshot_from_2024-11-06_10-51-48.png
Screenshot_from_2024-11-06_10-51-48.png
Attachment 11: Screenshot_from_2024-11-06_10-52-49.png
Screenshot_from_2024-11-06_10-52-49.png
Attachment 12: Screenshot_from_2024-11-06_10-54-12.png
Screenshot_from_2024-11-06_10-54-12.png
Attachment 13: Screenshot_from_2024-11-06_10-55-07.png
Screenshot_from_2024-11-06_10-55-07.png
Attachment 14: Screenshot_from_2024-11-06_10-55-56.png
Screenshot_from_2024-11-06_10-55-56.png
Attachment 15: Screenshot_from_2024-11-06_11-01-47.png
Screenshot_from_2024-11-06_11-01-47.png
Attachment 16: Screenshot_from_2024-11-06_11-02-48.png
Screenshot_from_2024-11-06_11-02-48.png
Attachment 17: Screenshot_from_2024-11-06_11-04-21.png
Screenshot_from_2024-11-06_11-04-21.png
Attachment 18: Screenshot_from_2024-11-06_11-05-09.png
Screenshot_from_2024-11-06_11-05-09.png
Attachment 19: Screenshot_from_2024-11-06_11-06-16.png
Screenshot_from_2024-11-06_11-06-16.png
Attachment 20: Screenshot_from_2024-11-06_11-07-16.png
Screenshot_from_2024-11-06_11-07-16.png
Attachment 21: Screenshot_from_2024-11-06_11-08-07.png
Screenshot_from_2024-11-06_11-08-07.png
Attachment 22: Screenshot_from_2024-11-06_11-09-09.png
Screenshot_from_2024-11-06_11-09-09.png
Attachment 23: Screenshot_from_2024-11-08_03-36-38.png
Screenshot_from_2024-11-08_03-36-38.png
Attachment 24: Screenshot_from_2024-11-08_03-39-13.png
Screenshot_from_2024-11-08_03-39-13.png
  669   Wed Aug 14 12:40:11 2024 JBRepaired DSSSD delivery 14.09.2024

Three BB18-1000 triples AIDAs collected on 14.09.2024

Find attached visual of the wafer and bond wire + factory bias tests accompanying the DSSSDs. elog:669/1 elog:669/2 elog:669/3

Visual inspection carried out showed that bond wires have been fixed + fingerprint on one DSSSD removed and wires repaired. elog:669/4 elog:669/5 elog:669/6 elog:669/7 elog:669/8 elog:669/9 elog:669/10 elog:669/11

DSSSD 1 (defect bias issue 80V): 3208-10 / 3208-18 / 3208-20

DSSSD 2  (3208-6 dysfunctional): 3208-6 / 3208-9 / 3208-16

DSSSD 3 (defect fingerprint): 3131-5 / 3131-10 / 3131-12

 

Attachment 1: doc00418120240814132739.pdf
doc00418120240814132739.pdf
Attachment 2: doc00418020240814132419.pdf
doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf
Attachment 3: doc00417920240814132325.pdf
doc00417920240814132325.pdf doc00417920240814132325.pdf doc00417920240814132325.pdf
Attachment 4: 20240814_132617.jpg
20240814_132617.jpg
Attachment 5: 20240814_133316.jpg
20240814_133316.jpg
Attachment 6: 20240814_133306.jpg
20240814_133306.jpg
Attachment 7: 20240814_133225.jpg
20240814_133225.jpg
Attachment 8: 20240814_133321.jpg
20240814_133321.jpg
Attachment 9: 20240814_133328.jpg
20240814_133328.jpg
Attachment 10: 20240814_133300.jpg
20240814_133300.jpg
Attachment 11: 20240814_133313.jpg
20240814_133313.jpg
  668   Sun Jul 14 16:52:28 2024 TDS181 R7_38-44
Attachment 1: R7_44.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257387880 ( 1999547.7 Hz)
        Other data format:    4532120 (   35208.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1033 (       8.0 Hz)
                               RESUME:       1034 (       8.0 Hz)
                              SYNC100:      32479 (     252.3 Hz)
                              WR48-63:      32479 (     252.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4465095 (   34687.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     632762 (    4915.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      128.723 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.236                0.000
  1                1.555                0.000
  2                0.113                0.000
  3                0.947                0.000
  4                0.002                0.000
  5                1.755                0.000
  6                0.004                0.000
  7               10.333                0.000
  8                0.007                0.000
  9                2.936                0.000
 10                2.071                0.000
 11                0.002                0.000
 12                0.000                0.000
 13                0.489                0.000
 14                0.000                0.000
 15                1.734                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14350588       7653          0          0         30         30       1869       1869          0       3855          0      17361
  1   13486473       3480          0          0         46         46       1694       1694          0          0          0      37855
  2   14703665    1320670          0          0         15         15       2021       2021          0    1316598          0      23155
  3   19664164       4986          0          0         55         55       2438       2438          0          0          0      30308
  4    5751938     622359          0          0          1          1        800        800          0     620757          0      10363
  5   18754826       4878          0          0        135        135       2304       2304          0          0          0      35466
  6    5749940     466689          0          0          1          1        788        788          0     465111          0      22939
  7   43562502      11323          0          0        235        236       5426       5426          0          0          0     306189
  8    6498781       1588          0          0          3          3        791        791          0          0          0      27083
  9   35613051    1294837          0          0        123        123       4631       4631          0    1285329          0      25629
 10   18394694       5004          0          0        235        235       2267       2267          0          0          0      15231
 11    4718587     774761          0          0          1          1        657        657          0     773445          0      24458
 12    4562825       1124          0          0          0          0        562        562          0          0          0       6576
 13   14049104       3450          0          0         62         62       1663       1663          0          0          0      22623
 14    5429441       1318          0          0          0          0        659        659          0          0          0      16486
 15   32097301       8000          0          0         91         91       3909       3909          0          0          0      11040
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.094s ( 1099.893 blocks/s,  68.743 Mb/s)
Attachment 2: R7_43.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257380554 ( 1952632.9 Hz)
        Other data format:    4539446 (   34438.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1049 (       8.0 Hz)
                               RESUME:       1048 (       8.0 Hz)
                              SYNC100:      32491 (     246.5 Hz)
                              WR48-63:      32491 (     246.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4472367 (   33929.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     637947 (    4839.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      131.812 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.211                0.000
  1                0.644                0.000
  2                0.104                0.000
  3                0.610                0.000
  4                0.001                0.000
  5                1.489                0.000
  6                0.000                0.000
  7               10.407                0.000
  8                0.007                0.000
  9                3.903                0.000
 10                2.105                0.000
 11                0.015                0.000
 12                0.000                0.000
 13                0.359                0.000
 14                0.001                0.000
 15                2.207                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14064394       7614          0          0         30         30       1803       1803          0       3948          0      17314
  1   13179423       3388          0          0         29         29       1665       1665          0          0          0      38151
  2   14557381    1335568          0          0         13         13       1972       1972          0    1331598          0      23475
  3   19617646       4892          0          0         43         43       2403       2403          0          0          0      30324
  4    5648958     616805          0          0          1          1        777        777          0     615249          0      10380
  5   18696180       4932          0          0        131        131       2335       2335          0          0          0      35339
  6    5815232     464164          0          0          0          0        770        770          0     462624          0      23216
  7   44269814      11633          0          0        225        224       5592       5592          0          0          0     313471
  8    6429091       1516          0          0          1          1        757        757          0          0          0      26517
  9   35880468    1300754          0          0        152        152       4574       4574          0    1291302          0      24969
 10   18353794       4866          0          0        246        246       2187       2187          0          0          0      14856
 11    4621196     769080          0          0          4          4        713        713          0     767646          0      24241
 12    4572345       1024          0          0          0          0        512        512          0          0          0       6439
 13   13956451       3570          0          0         56         56       1729       1729          0          0          0      22520
 14    5345915       1318          0          0          1          1        658        658          0          0          0      15800
 15   32372266       8322          0          0        117        117       4044       4044          0          0          0      10935
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.129s ( 1098.565 blocks/s,  68.660 Mb/s)
Attachment 3: R7_42.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257357615 ( 1959593.4 Hz)
        Other data format:    4562385 (   34739.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1086 (       8.3 Hz)
                               RESUME:       1086 (       8.3 Hz)
                              SYNC100:      32490 (     247.4 Hz)
                              WR48-63:      32490 (     247.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4495233 (   34228.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     640512 (    4877.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      131.332 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.320                0.000
  1                0.555                0.000
  2                0.107                0.000
  3                1.008                0.000
  4                0.000                0.000
  5                1.900                0.000
  6                0.000                0.000
  7               10.018                0.000
  8                0.003                0.000
  9                3.911                0.000
 10                2.003                0.000
 11                0.016                0.000
 12                0.000                0.000
 13                0.450                0.000
 14                0.001                0.000
 15                1.832                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13873525       7404          0          0         41         41       1699       1699          0       3924          0      17674
  1   13645773       3570          0          0         26         26       1759       1759          0          0          0      38841
  2   14382089    1335288          0          0         15         15       1902       1902          0    1331454          0      23704
  3   19838847       5258          0          0         58         58       2571       2571          0          0          0      30478
  4    5609330     626216          0          0          1          1        780        780          0     624654          0      10429
  5   18732852       4878          0          0        132        132       2307       2307          0          0          0      35532
  6    5773741     470345          0          0          0          0        796        796          0     468753          0      23367
  7   44302756      11512          0          0        235        235       5521       5521          0          0          0     312043
  8    6624893       1678          0          0          1          1        838        838          0          0          0      26919
  9   35723194    1298446          0          0        149        149       4530       4530          0    1289088          0      25317
 10   18313583       5042          0          0        245        245       2276       2276          0          0          0      15157
 11    4596970     778626          0          0          5          5        628        628          0     777360          0      24418
 12    4585974       1066          0          0          0          0        533        533          0          0          0       6525
 13   13753094       3518          0          0         70         70       1689       1689          0          0          0      22808
 14    5336029       1368          0          0          1          1        683        683          0          0          0      16140
 15   32264965       8170          0          0        107        107       3978       3978          0          0          0      11160
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.191s ( 1096.213 blocks/s,  68.513 Mb/s)
Attachment 4: R7_41.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257421108 ( 1939002.8 Hz)
        Other data format:    4498892 (   33887.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1059 (       8.0 Hz)
                               RESUME:       1059 (       8.0 Hz)
                              SYNC100:      32494 (     244.8 Hz)
                              WR48-63:      32494 (     244.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4431786 (   33382.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     626091 (    4716.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      132.760 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.208                0.000
  1                0.426                0.000
  2                0.135                0.000
  3                0.540                0.000
  4                0.004                0.000
  5                1.529                0.000
  6                0.001                0.000
  7               11.561                0.000
  8                0.001                0.000
  9                3.166                0.000
 10                1.981                0.000
 11                0.010                0.000
 12                0.003                0.000
 13                0.435                0.000
 14                0.001                0.000
 15                2.192                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14125854       7603          0          0         34         34       1780       1780          0       3975          0      16932
  1   12999507       3346          0          0         27         27       1646       1646          0          0          0      37605
  2   14549105    1330849          0          0         16         16       1993       1993          0    1326831          0      22570
  3   19588079       5104          0          0         44         44       2508       2508          0          0          0      29463
  4    5623374     602489          0          0          1          1        771        771          0     600945          0      10031
  5   18635917       5070          0          0        145        145       2390       2390          0          0          0      34577
  6    5774011     454485          0          0          1          1        755        755          0     452973          0      22358
  7   43927148      11232          0          0        252        252       5364       5364          0          0          0     309324
  8    6481984       1536          0          0          1          1        767        767          0          0          0      25848
  9   36293972    1305814          0          0        134        134       4617       4617          0    1296312          0      24663
 10   18484729       5190          0          0        226        226       2369       2369          0          0          0      14628
 11    4592277     752114          0          0          2          2        680        680          0     750750          0      23479
 12    4565033       1142          0          0          2          2        569        569          0          0          0       6273
 13   13975352       3570          0          0         64         64       1721       1721          0          0          0      21912
 14    5319152       1192          0          0          1          1        595        595          0          0          0      15594
 15   32485614       8156          0          0        109        109       3969       3969          0          0          0      10834
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.234s ( 1094.602 blocks/s,  68.413 Mb/s)
Attachment 5: R7_40.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257446831 ( 1940911.1 Hz)
        Other data format:    4473169 (   33723.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1093 (       8.2 Hz)
                               RESUME:       1093 (       8.2 Hz)
                              SYNC100:      32494 (     245.0 Hz)
                              WR48-63:      32494 (     245.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4405995 (   33217.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     622088 (    4690.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      132.642 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.216                0.000
  1                0.343                0.000
  2                0.126                0.000
  3                0.759                0.000
  4                0.006                0.000
  5                1.457                0.000
  6                0.000                0.000
  7               11.340                0.000
  8                0.008                0.000
  9                3.704                0.000
 10                2.066                0.000
 11                0.009                0.000
 12                0.000                0.000
 13                0.422                0.000
 14                0.007                0.000
 15                1.901                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14192134       7643          0          0         32         32       1802       1802          0       3975          0      16831
  1   12692374       3340          0          0         21         21       1649       1649          0          0          0      37336
  2   14587574    1324651          0          0         22         22       2017       2017          0    1320573          0      22631
  3   19453349       5132          0          0         57         57       2509       2509          0          0          0      29511
  4    5663707     601044          0          0          2          2        715        715          0     599610          0       9909
  5   18656910       4898          0          0        117        117       2332       2332          0          0          0      34162
  6    5798018     454961          0          0          0          0        757        757          0     453447          0      22350
  7   44082007      11418          0          0        263        263       5446       5446          0          0          0     306534
  8    6461137       1572          0          0          4          4        782        782          0          0          0      25967
  9   36131717    1290933          0          0        150        150       4584       4584          0    1281465          0      24188
 10   18542819       5234          0          0        249        249       2368       2368          0          0          0      14717
 11    4581074     748195          0          0          3          3        632        632          0     746925          0      23542
 12    4576883       1108          0          0          0          0        554        554          0          0          0       6304
 13   14043798       3508          0          0         66         66       1688       1688          0          0          0      21907
 14    5310480       1328          0          0          2          2        662        662          0          0          0      15552
 15   32672850       8204          0          0        105        105       3997       3997          0          0          0      10647
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.266s ( 1093.433 blocks/s,  68.340 Mb/s)
Attachment 6: R7_39.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257666264 ( 1899576.9 Hz)
        Other data format:    4253736 (   31359.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1109 (       8.2 Hz)
                               RESUME:       1109 (       8.2 Hz)
                              SYNC100:      32506 (     239.6 Hz)
                              WR48-63:      32506 (     239.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4186506 (   30863.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     603733 (    4450.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      135.644 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.315                0.000
  1                0.465                0.000
  2                0.152                0.000
  3                0.477                0.000
  4                0.000                0.000
  5                1.410                0.000
  6                0.000                0.000
  7               10.830                0.000
  8                0.006                0.000
  9                3.516                0.000
 10                2.096                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.504                0.000
 14                0.001                0.000
 15                2.318                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14135604       7717          0          0         45         45       1787       1787          0       4053          0      15313
  1   11750051       2976          0          0         28         28       1460       1460          0          0          0      33988
  2   14645076    1299621          0          0         18         18       2082       2082          0    1295421          0      20497
  3   19028955       4938          0          0         42         42       2427       2427          0          0          0      26749
  4    5601003     542795          0          0          0          0        778        778          0     541239          0       8896
  5   18444566       4954          0          0        128        128       2349       2349          0          0          0      31140
  6    5705092     411813          0          0          1          1        713        713          0     410385          0      20283
  7   44751821      11668          0          0        247        247       5587       5587          0          0          0     317392
  8    6384929       1562          0          0          3          3        778        778          0          0          0      23462
  9   36840506    1271195          0          0        144        144       4618       4618          0    1261671          0      22142
 10   18869011       5166          0          0        245        245       2338       2338          0          0          0      13407
 11    4447340     674973          0          0          1          1        617        617          0     673737          0      21305
 12    4493746       1090          0          0          0          0        545        545          0          0          0       5562
 13   14105097       3536          0          0         80         80       1688       1688          0          0          0      19880
 14    5253775       1242          0          0          1          1        620        620          0          0          0      14101
 15   33209692       8490          0          0        126        126       4119       4119          0          0          0       9616
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.145s ( 1097.976 blocks/s,  68.624 Mb/s)
Attachment 7: R7_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258699123 ( 1711360.3 Hz)
        Other data format:    3220878 (   21306.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        926 (       6.1 Hz)
                               RESUME:        926 (       6.1 Hz)
                              SYNC100:      32563 (     215.4 Hz)
                              WR48-63:      32563 (     215.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3153900 (   20863.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     485537 (    3212.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      151.166 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.249                0.000
  1                0.120                0.000
  2                0.066                0.000
  3                0.354                0.000
  4                0.003                0.000
  5                0.536                0.000
  6                0.000                0.000
  7                6.425                0.000
  8                0.006                0.000
  9                3.101                0.000
 10                1.800                0.000
 11                0.004                0.000
 12                0.000                0.000
 13                0.331                0.000
 14                0.001                0.000
 15                1.796                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14033681       8147          0          0         36         36       1771       1771          0       4533          0       6714
  1    7323901       1916          0          0          9          9        949        949          0          0          0      15627
  2   14901799    1219051          0          0         10         10       2080       2080          0    1214871          0       9105
  3   17459053       4614          0          0         34         34       2273       2273          0          0          0      11881
  4    5291930     245124          0          0          1          1        734        734          0     243654          0       4222
  5   16613617       4290          0          0         76         76       2069       2069          0          0          0      14390
  6    5335939     196741          0          0          0          0        710        710          0     195321          0       9119
  7   48661913      12390          0          0        201        201       5994       5994          0          0          0     356859
  8    5678372       1424          0          0          2          2        710        710          0          0          0      10389
  9   40228814    1200218          0          0        144        144       5116       5116          0    1189698          0       9803
 10   20091293       5418          0          0        251        251       2458       2458          0          0          0       5747
 11    3741989     306779          0          0          1          1        477        477          0     305823          0       9660
 12    3923680        940          0          0          0          0        470        470          0          0          0       2649
 13   14468402       3682          0          0         54         54       1787       1787          0          0          0       8802
 14    4770705       1114          0          0          1          1        556        556          0          0          0       6164
 15   36174035       9030          0          0        106        106       4409       4409          0          0          0       4406
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.168s ( 1097.094 blocks/s,  68.568 Mb/s)
  667   Sun Jul 14 14:37:53 2024 TDS181 R6_375-380
Attachment 1: R6_380.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258540051 ( 2422222.2 Hz)
        Other data format:    3379949 (   31666.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1735 (      16.3 Hz)
                               RESUME:       1737 (      16.3 Hz)
                              SYNC100:      32396 (     303.5 Hz)
                              WR48-63:      32396 (     303.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3311685 (   31026.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     517373 (    4847.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      106.737 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.790                0.000
  1                8.947                0.000
  2                0.508                0.000
  3               19.756                0.000
  4                4.785                0.000
  5               20.920                0.000
  6                0.022                0.000
  7               25.352                0.000
  8                0.008                0.000
  9                3.185                0.000
 10                2.367                0.000
 11                0.358                0.000
 12                2.272                0.000
 13                0.389                0.000
 14                0.017                0.000
 15                2.304                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14698695       7082          0          0         57         57       1903       1903          0       3162          0      34241
  1   16000888       4254          0          0        104        104       2023       2023          0          0          0      42387
  2   14179598    1061970          0          0         24         24       1878       1878          0    1058166          0      16158
  3   21475534       6065          0          0        356        357       2676       2676          0          0          0      27190
  4   13248260     388594          0          0         86         86       1647       1647          0     385128          0      11106
  5   22463599       5966          0          0        228        228       2755       2755          0          0          0      27752
  6    5791719     322542          0          0          5          5        778        778          0     320976          0      14590
  7   35198362       9279          0          0        282        283       4357       4357          0          0          0     182456
  8    5823243       1532          0          0          2          2        764        764          0          0          0      15809
  9   28506851    1036505          0          0        134        134       3608       3608          0    1029021          0      13553
 10   15170339       4094          0          0        198        198       1849       1849          0          0          0       7980
 11    9668170     517738          0          0         28         28       1225       1225          0     515232          0      45139
 12   10627809       2786          0          0         97         97       1296       1296          0          0          0      10320
 13   13616594       3424          0          0         42         42       1670       1670          0          0          0      24397
 14    5270456       1294          0          0          5          5        642        642          0          0          0       8930
 15   26799934       6824          0          0         87         87       3325       3325          0          0          0      35365
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.965s ( 1067.918 blocks/s,  66.745 Mb/s)
Attachment 2: R6_379.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258492867 ( 2488518.1 Hz)
        Other data format:    3427133 (   32993.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1669 (      16.1 Hz)
                               RESUME:       1667 (      16.0 Hz)
                              SYNC100:      32386 (     311.8 Hz)
                              WR48-63:      32386 (     311.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3359025 (   32337.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     534041 (    5141.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      103.874 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.531                0.000
  1                9.105                0.000
  2                0.409                0.000
  3               20.901                0.000
  4                6.817                0.000
  5               21.213                0.000
  6                0.044                0.000
  7               25.524                0.000
  8                0.012                0.000
  9                2.155                0.000
 10                2.654                0.000
 11                0.254                0.000
 12                3.206                0.000
 13                0.405                0.000
 14                0.021                0.000
 15                1.675                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14465712       6849          0          0         46         46       1829       1829          0       3099          0      35543
  1   16450883       4372          0          0         97         97       2089       2089          0          0          0      45383
  2   14055542    1052711          0          0         21         21       1792       1792          0    1049085          0      16931
  3   21598001       6069          0          0        353        352       2682       2682          0          0          0      28874
  4   13375374     396382          0          0         93         93       1673       1673          0     392850          0      11316
  5   23169470       6376          0          0        222        222       2966       2966          0          0          0      29587
  6    5868310     338835          0          0          6          6        756        756          0     337311          0      15349
  7   34787878       9109          0          0        284        283       4271       4271          0          0          0     179444
  8    5913627       1550          0          0          4          4        771        771          0          0          0      16483
  9   28000125    1037778          0          0         94         94       3590       3590          0    1030410          0      14411
 10   14658992       4034          0          0        206        206       1811       1811          0          0          0       8453
 11    9978683     548866          0          0         21         21       1277       1277          0     546270          0      48576
 12   11124676       2942          0          0        108        108       1363       1363          0          0          0      10622
 13   13313729       3352          0          0         38         38       1638       1638          0          0          0      25010
 14    5305479       1290          0          0          7          7        638        638          0          0          0       9613
 15   26426386       6618          0          0         69         69       3240       3240          0          0          0      38446
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.004s ( 1066.528 blocks/s,  66.658 Mb/s)
Attachment 3: R6_378.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258502011 ( 2458827.2 Hz)
        Other data format:    3417989 (   32511.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1749 (      16.6 Hz)
                               RESUME:       1749 (      16.6 Hz)
                              SYNC100:      32392 (     308.1 Hz)
                              WR48-63:      32392 (     308.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3349707 (   31861.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     532400 (    5064.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      105.132 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.679                0.000
  1                9.249                0.000
  2                0.786                0.000
  3               21.653                0.000
  4                5.446                0.000
  5               20.569                0.000
  6                0.018                0.000
  7               25.965                0.000
  8                0.023                0.000
  9                3.527                0.000
 10                3.084                0.000
 11                0.578                0.000
 12                2.211                0.000
 13                0.401                0.000
 14                0.029                0.000
 15                2.039                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14837825       7065          0          0         54         54       1908       1908          0       3141          0      35149
  1   15971414       4190          0          0         90         90       2005       2005          0          0          0      44340
  2   13973975    1055388          0          0         28         28       1877       1877          0    1051578          0      16603
  3   21360293       6062          0          0        359        359       2672       2672          0          0          0      27349
  4   13445710     405480          0          0         78         78       1722       1722          0     401880          0      11761
  5   22822103       6008          0          0        219        219       2785       2785          0          0          0      29227
  6    5869439     336469          0          0          3          3        764        764          0     334935          0      15220
  7   34904209       9014          0          0        288        288       4219       4219          0          0          0     183752
  8    5764878       1344          0          0          3          3        669        669          0          0          0      16353
  9   28081264    1028257          0          0        128        128       3666       3666          0    1020669          0      13917
 10   14858837       4056          0          0        228        228       1800       1800          0          0          0       8182
 11   10178261     540198          0          0         44         44       1303       1303          0     537504          0      47495
 12   10803258       2976          0          0        104        104       1384       1384          0          0          0      11101
 13   13631305       3368          0          0         38         38       1646       1646          0          0          0      24976
 14    5355037       1370          0          0         10         10        675        675          0          0          0       9380
 15   26644203       6744          0          0         75         75       3297       3297          0          0          0      37595
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.863s ( 1036.831 blocks/s,  64.802 Mb/s)
Attachment 4: R6_377.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258552134 ( 2473392.2 Hz)
        Other data format:    3367866 (   32218.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1756 (      16.8 Hz)
                               RESUME:       1758 (      16.8 Hz)
                              SYNC100:      32389 (     309.8 Hz)
                              WR48-63:      32389 (     309.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3299574 (   31564.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527301 (    5044.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      104.533 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.597                0.000
  1                8.636                0.000
  2                0.904                0.000
  3               20.863                0.000
  4                6.343                0.000
  5               20.105                0.000
  6                0.078                0.000
  7               25.706                0.000
  8                0.032                0.000
  9                3.757                0.000
 10                2.723                0.000
 11                0.441                0.000
 12                2.624                0.000
 13                0.350                0.000
 14                0.041                0.000
 15                2.134                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14829940       6930          0          0         43         43       1859       1859          0       3126          0      34968
  1   16084855       4240          0          0         86         86       2034       2034          0          0          0      44487
  2   14095291    1043346          0          0         25         25       1901       1901          0    1039494          0      16177
  3   21497704       5946          0          0        351        351       2622       2622          0          0          0      27780
  4   13645755     392320          0          0         90         90       1745       1745          0     388650          0      11516
  5   22699356       6082          0          0        212        212       2829       2829          0          0          0      28940
  6    5799289     329603          0          0          9          9        757        757          0     328071          0      14660
  7   34611640       9059          0          0        283        284       4246       4246          0          0          0     182416
  8    5904403       1412          0          0          5          5        701        701          0          0          0      16059
  9   27962268    1015369          0          0        138        139       3606       3606          0    1007880          0      13650
 10   14955138       4224          0          0        234        234       1878       1878          0          0          0       8242
 11    9968144     534917          0          0         35         35       1247       1247          0     532353          0      46585
 12   11007856       2816          0          0        109        109       1299       1299          0          0          0      10892
 13   13574135       3480          0          0         32         32       1708       1708          0          0          0      24567
 14    5407517       1306          0          0          8          8        645        645          0          0          0       9376
 15   26508843       6816          0          0         96         96       3312       3312          0          0          0      36986
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.332s ( 1054.990 blocks/s,  65.937 Mb/s)
Attachment 5: R6_376.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258492402 ( 2497580.9 Hz)
        Other data format:    3427598 (   33117.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1769 (      17.1 Hz)
                               RESUME:       1767 (      17.1 Hz)
                              SYNC100:      32385 (     312.9 Hz)
                              WR48-63:      32385 (     312.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3359292 (   32457.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     535580 (    5174.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      103.497 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.776                0.000
  1                9.861                0.000
  2                0.678                0.000
  3               21.902                0.000
  4                6.683                0.000
  5               20.300                0.000
  6                0.016                0.000
  7               26.525                0.000
  8                0.027                0.000
  9                3.024                0.000
 10                2.649                0.000
 11                0.458                0.000
 12                2.900                0.000
 13                0.387                0.000
 14                0.048                0.000
 15                2.584                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14764877       6954          0          0         56         56       1876       1876          0       3090          0      36135
  1   16480843       4336          0          0        118        118       2050       2050          0          0          0      45658
  2   14252286    1048007          0          0         23         23       1904       1904          0    1044153          0      16692
  3   21564846       6026          0          0        350        350       2663       2663          0          0          0      28647
  4   13304736     404308          0          0         88         88       1702       1702          0     400728          0      11809
  5   23013714       6376          0          0        231        231       2957       2957          0          0          0      30027
  6    5873061     342063          0          0          3          3        780        780          0     340497          0      15123
  7   34298981       8993          0          0        299        298       4198       4198          0          0          0     178104
  8    5866367       1500          0          0          3          3        747        747          0          0          0      16863
  9   27866733    1026338          0          0        122        121       3511       3511          0    1019073          0      14551
 10   14765796       4022          0          0        190        190       1821       1821          0          0          0       8481
 11   10252313     554457          0          0         32         32       1321       1321          0     551751          0      48407
 12   11025718       2958          0          0        115        115       1364       1364          0          0          0      11098
 13   13565855       3326          0          0         35         35       1628       1628          0          0          0      26079
 14    5404814       1304          0          0         10         10        642        642          0          0          0       9715
 15   26191462       6630          0          0         94         94       3221       3221          0          0          0      38191
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.238s ( 1058.261 blocks/s,  66.141 Mb/s)
Attachment 6: R6_375.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258498122 ( 2532033.6 Hz)
        Other data format:    3421878 (   33517.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1769 (      17.3 Hz)
                               RESUME:       1769 (      17.3 Hz)
                              SYNC100:      32380 (     317.2 Hz)
                              WR48-63:      32380 (     317.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3353580 (   32848.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     533348 (    5224.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      102.091 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.722                0.000
  1                9.219                0.000
  2                0.540                0.000
  3               21.422                0.000
  4                4.718                0.000
  5               21.434                0.000
  6                0.023                0.000
  7               26.932                0.000
  8                0.014                0.000
  9                3.140                0.000
 10                2.560                0.000
 11                0.453                0.000
 12                2.280                0.000
 13                0.408                0.000
 14                0.012                0.000
 15                2.073                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14893366       6777          0          0         60         60       1809       1809          0       3039          0      36151
  1   16620608       4354          0          0        112        112       2065       2065          0          0          0      45709
  2   13854193    1042012          0          0         29         29       1893       1893          0    1038168          0      16872
  3   21513605       6166          0          0        352        352       2731       2731          0          0          0      28236
  4   13835719     417225          0          0         86         86       1762       1762          0     413529          0      11794
  5   22712864       6078          0          0        213        213       2826       2826          0          0          0      29593
  6    5842394     341982          0          0          7          7        740        740          0     340488          0      15332
  7   33822775       8894          0          0        271        271       4176       4176          0          0          0     176325
  8    5896518       1406          0          0          3          3        700        700          0          0          0      16319
  9   27703752    1015898          0          0        111        111       3577       3577          0    1008522          0      13968
 10   14877004       4052          0          0        234        234       1792       1792          0          0          0       8511
 11   10500301     552528          0          0         30         30       1317       1317          0     549834          0      49042
 12   11072905       2914          0          0        119        119       1338       1338          0          0          0      10998
 13   13651284       3526          0          0         44         44       1719       1719          0          0          0      25633
 14    5433744       1396          0          0          4          4        694        694          0          0          0       9657
 15   26267090       6670          0          0         94         94       3241       3241          0          0          0      39208
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.184s ( 1060.179 blocks/s,  66.261 Mb/s)
  666   Mon Jun 17 02:13:21 2024 TDMonday 17 June
03.11 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4

05.33 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-11
       aida09 pulser walkthrough 60 ch FWHM

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-14

      Merger, TapeServer etc - attachments 15-16

05.40 analysis data file R9_1042 - attachment 17

      Check ASIC load x2

      aida02 zero rate
      restart Merger

05.55 Data file S181/R10
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 25Hz

      per p+n FEE64 1.8.L spectra - attachment 18

      per FEE64 Rate spectra - attachment 19

06.43 TapeServer no storage mode

09.14 Data file S181/R11
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 25Hz

09.52 TapeServer no storage mode

      per n+n FEE64 1.8.L spectra - attachment 20

15.18 DEFGAS/FATIMA arrays moved downstream to permit installation of calibration source at centre of each array - increased distance to AIDA/bPlas => lower rate


15.16 DSSSD bias & leakage current OK - attachments 21-22

      FEE64 temperatures OK - attachment 23

      ADC data item stats OK - attachment 24
       aida02 zero rate

      per FEE64 Rate spectra - attachment 25

      Merger restart
      ADC data item stats OK - attachment 26
       7x < 20k, max 207k aida16

15.54 bPlast debiased.


16.29 Data file S181/R12
      bPlas OFF but remiains cabled/connected/grounded

16.30 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30

      per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - see attachments 31-39
       note - not all FEE64s updated spectra for all settings

      ADC data item stats OK - attachment 40
       rates change cf. attachment 29
       sources installed/moved/removed, people moving around platform?

      per FEE64 Rate spectra - attachment 41

16.57 DSSSD bias OFF
      FEE64 power OFF

16.34 analysis data file S181/R12_7
      max deadtime 2.2% (aida08)
      ADC data rate 1.330M, HEC data rate 2.1k

16.52 analysis data file S181/R12_12
      max deadtime 1.7% (aida08)
      ADC data rate 945k, HEC data rate 1.8k
Attachment 1: Screenshot_from_2024-06-17_03-11-38.png
Screenshot_from_2024-06-17_03-11-38.png
Attachment 2: Screenshot_from_2024-06-17_03-12-05.png
Screenshot_from_2024-06-17_03-12-05.png
Attachment 3: Screenshot_from_2024-06-17_03-12-29.png
Screenshot_from_2024-06-17_03-12-29.png
Attachment 4: Screenshot_from_2024-06-17_03-12-53.png
Screenshot_from_2024-06-17_03-12-53.png
Attachment 5: Screenshot_from_2024-06-17_05-33-18.png
Screenshot_from_2024-06-17_05-33-18.png
Attachment 6: Screenshot_from_2024-06-17_05-33-47.png
Screenshot_from_2024-06-17_05-33-47.png
Attachment 7: Screenshot_from_2024-06-17_05-34-22.png
Screenshot_from_2024-06-17_05-34-22.png
Attachment 8: Screenshot_from_2024-06-17_05-34-53.png
Screenshot_from_2024-06-17_05-34-53.png
Attachment 9: Screenshot_from_2024-06-17_05-36-26.png
Screenshot_from_2024-06-17_05-36-26.png
Attachment 10: Screenshot_from_2024-06-17_05-38-40.png
Screenshot_from_2024-06-17_05-38-40.png
Attachment 11: Screenshot_from_2024-06-17_05-39-53.png
Screenshot_from_2024-06-17_05-39-53.png
Attachment 12: Screenshot_from_2024-06-17_05-40-29.png
Screenshot_from_2024-06-17_05-40-29.png
Attachment 13: Screenshot_from_2024-06-17_05-41-12.png
Screenshot_from_2024-06-17_05-41-12.png
Attachment 14: Screenshot_from_2024-06-17_05-42-07.png
Screenshot_from_2024-06-17_05-42-07.png
Attachment 15: Screenshot_from_2024-06-17_05-42-35.png
Screenshot_from_2024-06-17_05-42-35.png
Attachment 16: Screenshot_from_2024-06-17_05-43-06.png
Screenshot_from_2024-06-17_05-43-06.png
Attachment 17: R9_1042
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259119494 ( 1239652.5 Hz)
        Other data format:    2800506 (   13397.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        509 (       2.4 Hz)
                               RESUME:        509 (       2.4 Hz)
                              SYNC100:      32779 (     156.8 Hz)
                              WR48-63:      32779 (     156.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2733930 (   13079.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     425638 (    2036.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      209.026 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.013                0.000
  1                0.000                0.000
  2                0.006                0.000
  3                0.367                0.000
  4                0.000                0.000
  5                0.343                0.000
  6                0.000                0.000
  7                4.268                0.000
  8                0.000                0.000
  9                2.321                0.000
 10                0.126                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.030                0.000
 14                0.000                0.000
 15                1.478                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9607668       8795          0          0         21         21       1240       1240          0       6273          0          0
  1     568341        148          0          0          0          0         74         74          0          0          0          0
  2   11592594    1368472          0          0          2          2       1710       1710          0    1365048          0          0
  3   20057241       5226          0          0         32         32       2581       2581          0          0          0          0
  4    4038921       1042          0          0          0          0        521        521          0          0          0          0
  5   16834727       4288          0          0         27         27       2117       2117          0          0          0          0
  6    2895864      13265          0          0          0          0        361        361          0      12543          0          0
  7   58542191      15052          0          0        171        171       7355       7355          0          0          0     425638
  8    6363514       1604          0          0          0          0        802        802          0          0          0          0
  9   47066112    1362366          0          0        140        140       6010       6010          0    1350066          0          0
 10   19083458       4678          0          0         24         24       2315       2315          0          0          0          0
 11    2437006        602          0          0          0          0        301        301          0          0          0          0
 12    3839787        956          0          0          0          0        478        478          0          0          0          0
 13   12055396       3054          0          0         12         12       1515       1515          0          0          0          0
 14    3699941        872          0          0          0          0        436        436          0          0          0          0
 15   40436733      10086          0          0         80         80       4963       4963          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   69.695s (  459.141 blocks/s,  28.696 Mb/s)
Attachment 18: Screenshot_from_2024-06-17_06-44-16.png
Screenshot_from_2024-06-17_06-44-16.png
Attachment 19: Screenshot_from_2024-06-17_06-45-49.png
Screenshot_from_2024-06-17_06-45-49.png
Attachment 20: Screenshot_from_2024-06-17_09-53-53.png
Screenshot_from_2024-06-17_09-53-53.png
Attachment 21: Screenshot_from_2024-06-17_15-17-20.png
Screenshot_from_2024-06-17_15-17-20.png
Attachment 22: Screenshot_from_2024-06-17_15-16-47.png
Screenshot_from_2024-06-17_15-16-47.png
Attachment 23: Screenshot_from_2024-06-17_15-17-46.png
Screenshot_from_2024-06-17_15-17-46.png
Attachment 24: Screenshot_from_2024-06-17_15-18-09.png
Screenshot_from_2024-06-17_15-18-09.png
Attachment 25: Screenshot_from_2024-06-17_15-18-54.png
Screenshot_from_2024-06-17_15-18-54.png
Attachment 26: Screenshot_from_2024-06-17_15-20-50.png
Screenshot_from_2024-06-17_15-20-50.png
Attachment 27: Screenshot_from_2024-06-17_16-30-26.png
Screenshot_from_2024-06-17_16-30-26.png
Attachment 28: Screenshot_from_2024-06-17_16-30-55.png
Screenshot_from_2024-06-17_16-30-55.png
Attachment 29: Screenshot_from_2024-06-17_16-31-26.png
Screenshot_from_2024-06-17_16-31-26.png
Attachment 30: Screenshot_from_2024-06-17_16-31-48.png
Screenshot_from_2024-06-17_16-31-48.png
Attachment 31: Screenshot_from_2024-06-17_16-33-21.png
Screenshot_from_2024-06-17_16-33-21.png
Attachment 32: Screenshot_from_2024-06-17_16-34-32.png
Screenshot_from_2024-06-17_16-34-32.png
Attachment 33: Screenshot_from_2024-06-17_16-36-35.png
Screenshot_from_2024-06-17_16-36-35.png
Attachment 34: Screenshot_from_2024-06-17_16-37-33.png
Screenshot_from_2024-06-17_16-37-33.png
Attachment 35: Screenshot_from_2024-06-17_16-38-14.png
Screenshot_from_2024-06-17_16-38-14.png
Attachment 36: Screenshot_from_2024-06-17_16-40-21.png
Screenshot_from_2024-06-17_16-40-21.png
Attachment 37: Screenshot_from_2024-06-17_16-41-18.png
Screenshot_from_2024-06-17_16-41-18.png
Attachment 38: Screenshot_from_2024-06-17_16-43-19.png
Screenshot_from_2024-06-17_16-43-19.png
Attachment 39: Screenshot_from_2024-06-17_16-44-24.png
Screenshot_from_2024-06-17_16-44-24.png
Attachment 40: Screenshot_from_2024-06-17_16-45-19.png
Screenshot_from_2024-06-17_16-45-19.png
Attachment 41: Screenshot_from_2024-06-17_16-46-32.png
Screenshot_from_2024-06-17_16-46-32.png
Attachment 42: R12_7
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258100864 ( 1330189.9 Hz)
        Other data format:    3819136 (   19682.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        522 (       2.7 Hz)
                               RESUME:        522 (       2.7 Hz)
                              SYNC100:      32723 (     168.6 Hz)
                              WR48-63:      32723 (     168.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3752646 (   19340.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     409704 (    2111.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      194.033 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.121                0.000
  1                0.000                0.000
  2                0.050                0.000
  3                0.191                0.000
  4                0.010                0.000
  5                0.209                0.000
  6                0.000                0.000
  7                4.225                0.000
  8                0.000                0.000
  9                2.072                0.000
 10                0.065                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.145                0.000
 14                0.002                0.000
 15                1.159                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13428500       9208          0          0         38         38       1659       1659          0       5814          0          0
  1    4595656       1164          0          0          0          0        582        582          0          0          0          0
  2   13843408    1881338          0          0         11         11       1937       1937          0    1877442          0          0
  3   19769270       5066          0          0         33         33       2500       2500          0          0          0          0
  4    5034345       1228          0          0          4          4        610        610          0          0          0          0
  5   19195180       4990          0          0         40         40       2455       2455          0          0          0          0
  6    5179175      12949          0          0          0          0        653        653          0      11643          0          0
  7   53830866      13806          0          0        173        173       6730       6730          0          0          0     409704
  8    6018644       1564          0          0          0          0        782        782          0          0          0          0
  9   41936601    1868943          0          0        126        126       5472       5472          0    1857747          0          0
 10   16378984       4032          0          0         16         16       2000       2000          0          0          0          0
 11    2633344        646          0          0          0          0        323        323          0          0          0          0
 12    1918841        428          0          0          0          0        214        214          0          0          0          0
 13   12464437       2916          0          0         14         14       1444       1444          0          0          0          0
 14    3882711       1026          0          0          1          1        512        512          0          0          0          0
 15   37990902       9832          0          0         66         66       4850       4850          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   60.555s (  528.448 blocks/s,  33.028 Mb/s)
Attachment 43: R12_12
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       7946
          ADC data format:   63712775 (  945168.0 Hz)
        Other data format:    1325235 (   19659.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        128 (       1.9 Hz)
                               RESUME:        128 (       1.9 Hz)
                              SYNC100:       8197 (     121.6 Hz)
                              WR48-63:       8197 (     121.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1308585 (   19412.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     119370 (    1770.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       67.409 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.043                0.000
  4                0.000                0.000
  5                0.026                0.000
  6                0.000                0.000
  7                1.174                0.000
  8                0.000                0.000
  9                0.504                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.224                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     896977       2260          0          0          0          0        119        119          0       2022          0          0
  1    1331618        334          0          0          0          0        167        167          0          0          0          0
  2    2033117     654263          0          0          0          0        346        346          0     653571          0          0
  3    6066546       1620          0          0         14         14        796        796          0          0          0          0
  4     632291        132          0          0          0          0         66         66          0          0          0          0
  5    5664311       1468          0          0         11         11        723        723          0          0          0          0
  6    1249421       4352          0          0          0          0        154        154          0       4044          0          0
  7   16999500       4406          0          0         59         59       2144       2144          0          0          0     119370
  8      43320         14          0          0          0          0          7          7          0          0          0          0
  9   12410393     652234          0          0         26         26       1617       1617          0     648948          0          0
 10    3213140        792          0          0          0          0        396        396          0          0          0          0
 11     278998         64          0          0          0          0         32         32          0          0          0          0
 12     433582        116          0          0          0          0         58         58          0          0          0          0
 13    1869315        430          0          0          0          0        215        215          0          0          0          0
 14     612732        128          0          0          0          0         64         64          0          0          0          0
 15    9977514       2622          0          0         18         18       1293       1293          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   14.254s (  557.461 blocks/s,  34.841 Mb/s)
  665   Sun Jun 16 03:11:27 2024 TDSunday 16 June
04.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4


07.52 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per p+n FEE64 1.8.L spectra - attachments 9-10
       aida09 pulser peak width 59 ch FWHM


15.01 DSSSD bias & leakage current OK - attachment 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats OK - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-15
       aida09 pulser peak width 59 ch FWHM
       aida05 pulser peak width 68 ch FWHM
       aida12 pulser peak width 71 ch FWHM


      per FEE64 Rate spectra - attachment 16

      Merger, TapeSever etc - attachment 17
       no data transfer to MBS - MBS down?

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 


19.12 DSSSD bias & leakage current OK - attachment 19

      FEE64 temperatures OK - attachment 20

      ADC data item stats OK - attachment 21

      per FEE64 Rate spectra - attachment 22

23.34 DSSSD bias & leakage current OK - attachment 23

      FEE64 temperatures OK - attachment 24

      ADC data item stats OK - attachment 25

      per FEE64 Rate spectra - attachment 26
Attachment 1: Screenshot_from_2024-06-16_04-10-35.png
Screenshot_from_2024-06-16_04-10-35.png
Attachment 2: Screenshot_from_2024-06-16_04-09-07.png
Screenshot_from_2024-06-16_04-09-07.png
Attachment 3: Screenshot_from_2024-06-16_04-09-41.png
Screenshot_from_2024-06-16_04-09-41.png
Attachment 4: Screenshot_from_2024-06-16_04-10-21.png
Screenshot_from_2024-06-16_04-10-21.png
Attachment 5: Screenshot_from_2024-06-16_07-52-49.png
Screenshot_from_2024-06-16_07-52-49.png
Attachment 6: Screenshot_from_2024-06-16_07-53-20.png
Screenshot_from_2024-06-16_07-53-20.png
Attachment 7: Screenshot_from_2024-06-16_07-53-47.png
Screenshot_from_2024-06-16_07-53-47.png
Attachment 8: Screenshot_from_2024-06-16_07-54-42.png
Screenshot_from_2024-06-16_07-54-42.png
Attachment 9: Screenshot_from_2024-06-16_07-59-28.png
Screenshot_from_2024-06-16_07-59-28.png
Attachment 10: Screenshot_from_2024-06-16_08-01-35.png
Screenshot_from_2024-06-16_08-01-35.png
Attachment 11: Screenshot_from_2024-06-16_15-01-23.png
Screenshot_from_2024-06-16_15-01-23.png
Attachment 12: Screenshot_from_2024-06-16_15-01-46.png
Screenshot_from_2024-06-16_15-01-46.png
Attachment 13: Screenshot_from_2024-06-16_15-02-12.png
Screenshot_from_2024-06-16_15-02-12.png
Attachment 14: Screenshot_from_2024-06-16_15-03-05.png
Screenshot_from_2024-06-16_15-03-05.png
Attachment 15: Screenshot_from_2024-06-16_15-06-14.png
Screenshot_from_2024-06-16_15-06-14.png
Attachment 16: Screenshot_from_2024-06-16_15-07-06.png
Screenshot_from_2024-06-16_15-07-06.png
Attachment 17: Screenshot_from_2024-06-16_15-08-02.png
Screenshot_from_2024-06-16_15-08-02.png
Attachment 18: R9_747
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261839349 ( 1667447.6 Hz)
        Other data format:      80651 (     513.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        683 (       4.3 Hz)
                               RESUME:        683 (       4.3 Hz)
                              SYNC100:      32585 (     207.5 Hz)
                              WR48-63:      32585 (     207.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      14115 (      89.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     357749 (    2278.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      157.030 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.474                0.000
  1                0.000                0.000
  2                0.114                0.000
  3                0.086                0.000
  4                0.007                0.000
  5                0.165                0.000
  6                0.000                0.000
  7                6.815                0.000
  8                0.000                0.000
  9                3.068                0.000
 10                0.526                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.247                0.000
 14                0.002                0.000
 15                1.373                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17385482       9389          0          0         56         56       2291       2291          0       4695          0          0
  1    4359471       1110          0          0          0          0        555        555          0          0          0          0
  2   16890768       4312          0          0         14         14       2142       2142          0          0          0          0
  3   16799432       4226          0          0         17         17       2096       2096          0          0          0          0
  4    5895210       1414          0          0          2          2        705        705          0          0          0          0
  5   15605735       4074          0          0         27         27       2010       2010          0          0          0          0
  6    2976504      10096          0          0          0          0        338        338          0       9420          0          0
  7   48753528      12540          0          0        223        223       6047       6047          0          0          0     357748
  8    5419325       1414          0          0          0          0        707        707          0          0          0          0
  9   41464320      10520          0          0        151        151       5109       5109          0          0          0          0
 10   21531781       5458          0          0         66         66       2663       2663          0          0          0          1
 11    3242993        744          0          0          0          0        372        372          0          0          0          0
 12    3988909        970          0          0          0          0        485        485          0          0          0          0
 13   16192913       4074          0          0         38         38       1999       1999          0          0          0          0
 14    4620379       1122          0          0          1          1        560        560          0          0          0          0
 15   36712599       9188          0          0         88         88       4506       4506          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   63.340s (  505.211 blocks/s,  31.576 Mb/s)
Attachment 19: Screenshot_from_2024-06-16_19-11-54.png
Screenshot_from_2024-06-16_19-11-54.png
Attachment 20: Screenshot_from_2024-06-16_19-12-22.png
Screenshot_from_2024-06-16_19-12-22.png
Attachment 21: Screenshot_from_2024-06-16_19-14-40.png
Screenshot_from_2024-06-16_19-14-40.png
Attachment 22: Screenshot_from_2024-06-16_19-15-10.png
Screenshot_from_2024-06-16_19-15-10.png
Attachment 23: Screenshot_from_2024-06-16_23-34-39.png
Screenshot_from_2024-06-16_23-34-39.png
Attachment 24: Screenshot_from_2024-06-16_23-35-06.png
Screenshot_from_2024-06-16_23-35-06.png
Attachment 25: Screenshot_from_2024-06-16_23-35-37.png
Screenshot_from_2024-06-16_23-35-37.png
Attachment 26: Screenshot_from_2024-06-16_23-36-04.png
Screenshot_from_2024-06-16_23-36-04.png
  664   Sat Jun 15 00:28:01 2024 TDSaturday 15 June
Beam lost due to septum failure c. 18.15 Friday 14 June

Cooling of recirculating water failed c. 19.00 - water temperature increased from the normal 17-18deg C to 25-26 deg C
FEE64 temperatures increased and FEE64s powered off and on-call engineer requested.

Cooling of recirculating water restored and AIDA restarted

NH started background run 23.37 - data file S181/R9

01.21 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3
       8x < 20k, max 238k aida08

      per FEE64 Rate spectra - attachment 4

      2x Check ASIC load

      Merger, TapeServer etc - attachments 5-6

      analysis data file R9_27 - attachment 7
       max deadtime 1.9%, all other FEE64s < 1%
       ADC data rate 951k, HEC data rate 1.6kHz

04.28 DSSSD bias & leakage current OK - attachment 8

      FEE64 temperatures OK - attachment 9

      ADC data item stats OK - attachment 10
       8x < 20k, max 261k aida08

      per FEE64 Rate spectra - attachment 11


08.30 DSSSD bias & leakage current OK - attachments 12-13

      FEE64 temperatures OK - attachment 14

      ADC data item stats OK - attachment 15
       all rates increased - gamma source in use?

      per FEE64 Rate spectra - attachments 16-17
       note aida08 asic #3 hot HEC channel

      per FEE64 1.8.L spectra - attachments 18-20
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-23

      Merger, TapeServer etc - attachments 25-25

      analysis data file R9_152 - attachment 26
       max deadtime 2.4%, all other FEE64s < 1%
       ADC data rate 1.215M, HEC data rate 2.1kHz


12.22 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30



15.27 DSSSD bias & leakage current OK - attachment 31

      FEE64 temperatures OK - attachment 32

      ADC data item stats OK - attachment 33
       all rates increased - 370kBq 152Eu in use

      per FEE64 Rate spectra - attachments 34

      per FEE64 1.8.L spectra - attachments 35-36
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      analysis data file R9_275 - attachment 37
       max deadtime 4.4%, all other FEE64s < 2%
       ADC data rate 1.520M, HEC data rate 2.3kHz

18.52 DSSSD bias & leakage current OK - attachment 38

      FEE64 temperatures OK - attachment 39

      ADC data item stats OK - attachment 40

      per FEE64 Rate spectra - attachment 41


22.50 DSSSD bias & leakage current OK - attachment 42

      FEE64 temperatures OK - attachment 43

      ADC data item stats OK - attachment 44

      per FEE64 Rate spectra - attachment 45
Attachment 1: Screenshot_from_2024-06-15_01-21-06.png
Screenshot_from_2024-06-15_01-21-06.png
Attachment 2: Screenshot_from_2024-06-15_01-21-41.png
Screenshot_from_2024-06-15_01-21-41.png
Attachment 3: Screenshot_from_2024-06-15_01-23-59.png
Screenshot_from_2024-06-15_01-23-59.png
Attachment 4: Screenshot_from_2024-06-15_01-24-56.png
Screenshot_from_2024-06-15_01-24-56.png
Attachment 5: Screenshot_from_2024-06-15_01-27-07.png
Screenshot_from_2024-06-15_01-27-07.png
Attachment 6: Screenshot_from_2024-06-15_01-27-34.png
Screenshot_from_2024-06-15_01-27-34.png
Attachment 7: R9_27
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258219194 (  950585.8 Hz)
        Other data format:    3700806 (   13623.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        516 (       1.9 Hz)
                               RESUME:        516 (       1.9 Hz)
                              SYNC100:      33012 (     121.5 Hz)
                              WR48-63:      33012 (     121.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3633750 (   13377.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     437157 (    1609.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      271.642 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.523                0.000
  1                0.000                0.000
  2                0.959                0.000
  3                0.770                0.000
  4                0.002                0.000
  5                0.768                0.000
  6                0.000                0.000
  7                5.278                0.000
  8                0.000                0.000
  9                2.951                0.000
 10                0.780                0.000
 11                0.000                0.000
 12                0.030                0.000
 13                0.033                0.000
 14                0.000                0.000
 15                2.166                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3263639       8999          0          0          7          7        426        426          0       8133          0        129
  1    5814210       1456          0          0          0          0        728        728          0          0          0        128
  2    6330359    1813504          0          0          6          6       1067       1067          0    1811358          0         79
  3   25849187       6768          0          0         50         50       3334       3334          0          0          0        112
  4    2729875        722          0          0          1          1        360        360          0          0          0        127
  5   18600203       4844          0          0         37         37       2385       2385          0          0          0         94
  6    5217587      17698          0          0          0          0        698        698          0      16302          0        124
  7   65848180      17052          0          0        196        196       8330       8330          0          0          0     435453
  8    7897456       2060          0          0          0          0       1030       1030          0          0          0        128
  9   45422925    1809941          0          0        107        107       5885       5885          0    1797957          0         64
 10   15079549       3854          0          0         29         29       1898       1898          0          0          0        111
 11    1722257        384          0          0          0          0        192        192          0          0          0        128
 12    4556679       1158          0          0          1          1        578        578          0          0          0        141
 13    6725939       1702          0          0          2          2        849        849          0          0          0        130
 14    3055032        760          0          0          0          0        380        380          0          0          0        128
 15   40106117       9904          0          0         80         80       4872       4872          0          0          0         81
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   39.733s (  805.368 blocks/s,  50.335 Mb/s)
Attachment 8: Screenshot_from_2024-06-15_04-28-07.png
Screenshot_from_2024-06-15_04-28-07.png
Attachment 9: Screenshot_from_2024-06-15_04-28-40.png
Screenshot_from_2024-06-15_04-28-40.png
Attachment 10: Screenshot_from_2024-06-15_04-30-40.png
Screenshot_from_2024-06-15_04-30-40.png
Attachment 11: Screenshot_from_2024-06-15_04-31-38.png
Screenshot_from_2024-06-15_04-31-38.png
Attachment 12: Screenshot_from_2024-06-15_08-33-07.png
Screenshot_from_2024-06-15_08-33-07.png
Attachment 13: Screenshot_from_2024-06-15_08-30-26.png
Screenshot_from_2024-06-15_08-30-26.png
Attachment 14: Screenshot_from_2024-06-15_08-33-35.png
Screenshot_from_2024-06-15_08-33-35.png
Attachment 15: Screenshot_from_2024-06-15_08-36-46.png
Screenshot_from_2024-06-15_08-36-46.png
Attachment 16: Screenshot_from_2024-06-15_08-35-16.png
Screenshot_from_2024-06-15_08-35-16.png
Attachment 17: Screenshot_from_2024-06-15_08-43-12.png
Screenshot_from_2024-06-15_08-43-12.png
Attachment 18: Screenshot_from_2024-06-15_08-40-42.png
Screenshot_from_2024-06-15_08-40-42.png
Attachment 19: Screenshot_from_2024-06-15_08-39-42.png
Screenshot_from_2024-06-15_08-39-42.png
Attachment 20: Screenshot_from_2024-06-15_08-38-37.png
Screenshot_from_2024-06-15_08-38-37.png
Attachment 21: Screenshot_from_2024-06-15_08-46-26.png
Screenshot_from_2024-06-15_08-46-26.png
Attachment 22: Screenshot_from_2024-06-15_08-45-35.png
Screenshot_from_2024-06-15_08-45-35.png
Attachment 23: Screenshot_from_2024-06-15_08-44-50.png
Screenshot_from_2024-06-15_08-44-50.png
Attachment 24: Screenshot_from_2024-06-15_08-48-10.png
Screenshot_from_2024-06-15_08-48-10.png
Attachment 25: Screenshot_from_2024-06-15_08-47-40.png
Screenshot_from_2024-06-15_08-47-40.png
Attachment 26: R9_152
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259017680 ( 1215243.3 Hz)
        Other data format:    2902320 (   13616.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        515 (       2.4 Hz)
                               RESUME:        515 (       2.4 Hz)
                              SYNC100:      32794 (     153.9 Hz)
                              WR48-63:      32794 (     153.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2835702 (   13304.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     441097 (    2069.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      213.141 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.001                0.000
  1                0.000                0.000
  2                0.002                0.000
  3                0.140                0.000
  4                0.000                0.000
  5                0.047                0.000
  6                0.000                0.000
  7                5.152                0.000
  8                0.000                0.000
  9                2.136                0.000
 10                0.186                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.032                0.000
 14                0.000                0.000
 15                1.354                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8157992       8515          0          0          4          4       1057       1057          0       6393          0          0
  1    4152307       1016          0          0          0          0        508        508          0          0          0          0
  2   10764259    1418465          0          0          2          2       1556       1556          0    1415349          0          0
  3   20158462       5380          0          0         30         30       2660       2660          0          0          0          0
  4    3884729       1032          0          0          0          0        516        516          0          0          0          0
  5   16603608       4178          0          0         21         21       2068       2068          0          0          0          0
  6    5112596      13950          0          0          0          0        582        582          0      12786          0          0
  7   59646516      15586          0          0        217        217       7576       7576          0          0          0     441097
  8    6438302       1678          0          0          0          0        839        839          0          0          0          0
  9   45360093    1412938          0          0        126        126       5756       5756          0    1401174          0          0
 10   18847436       4734          0          0         36         36       2331       2331          0          0          0          0
 11    2364016        648          0          0          0          0        324        324          0          0          0          0
 12    1869299        460          0          0          0          0        230        230          0          0          0          0
 13   11268040       2624          0          0          8          8       1304       1304          0          0          0          0
 14    3774625        934          0          0          0          0        467        467          0          0          0          0
 15   40615400      10182          0          0         71         71       5020       5020          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.906s (  763.609 blocks/s,  47.726 Mb/s)
Attachment 27: Screenshot_from_2024-06-15_12-22-54.png
Screenshot_from_2024-06-15_12-22-54.png
Attachment 28: Screenshot_from_2024-06-15_12-23-19.png
Screenshot_from_2024-06-15_12-23-19.png
Attachment 29: Screenshot_from_2024-06-15_12-25-35.png
Screenshot_from_2024-06-15_12-25-35.png
Attachment 30: Screenshot_from_2024-06-15_12-26-47.png
Screenshot_from_2024-06-15_12-26-47.png
Attachment 31: Screenshot_from_2024-06-15_15-26-54.png
Screenshot_from_2024-06-15_15-26-54.png
Attachment 32: Screenshot_from_2024-06-15_15-27-27.png
Screenshot_from_2024-06-15_15-27-27.png
Attachment 33: Screenshot_from_2024-06-15_15-27-48.png
Screenshot_from_2024-06-15_15-27-48.png
Attachment 34: Screenshot_from_2024-06-15_15-28-25.png
Screenshot_from_2024-06-15_15-28-25.png
Attachment 35: Screenshot_from_2024-06-15_15-31-08.png
Screenshot_from_2024-06-15_15-31-08.png
Attachment 36: Screenshot_from_2024-06-15_15-32-10.png
Screenshot_from_2024-06-15_15-32-10.png
Attachment 37: R9_275
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259129992 ( 1519853.9 Hz)
        Other data format:    2790008 (   16364.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        665 (       3.9 Hz)
                               RESUME:        665 (       3.9 Hz)
                              SYNC100:      32636 (     191.4 Hz)
                              WR48-63:      32636 (     191.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2723406 (   15973.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     395024 (    2316.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.497 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.249                0.000
  1                0.000                0.000
  2                0.356                0.000
  3                0.136                0.000
  4                0.337                0.000
  5                0.141                0.000
  6                0.019                0.000
  7                7.476                0.000
  8                0.165                0.000
  9                3.059                0.000
 10                0.581                0.000
 11                0.000                0.000
 12                0.320                0.000
 13                0.705                0.000
 14                0.045                0.000
 15                1.921                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12819722       8373          0          0         35         35       1597       1597          0       5109          0          0
  1    4488999       1128          0          0          0          0        564        564          0          0          0          0
  2   13809452    1368881          0          0         15         15       1975       1975          0    1364901          0          0
  3   17302926       4370          0          0         25         25       2160       2160          0          0          0          0
  4    4688002       1104          0          0         11         11        541        541          0          0          0          0
  5   15844696       4176          0          0         32         32       2056       2056          0          0          0          0
  6    5051819      11648          0          0          1          1        708        708          0      10230          0          0
  7   52726699      13824          0          0        233        233       6679       6679          0          0          0     395024
  8    5506218       1384          0          0          2          2        690        690          0          0          0          0
  9   42499600    1354256          0          0        142        142       5403       5403          0    1343166          0          0
 10   20647689       5006          0          0         43         43       2460       2460          0          0          0          0
 11    2780744        604          0          0          0          0        302        302          0          0          0          0
 12    3628914        944          0          0          9          9        463        463          0          0          0          0
 13   14531072       3696          0          0         25         25       1823       1823          0          0          0          0
 14    4089821       1130          0          0          3          3        562        562          0          0          0          0
 15   38713619       9484          0          0         89         89       4653       4653          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   49.219s (  650.159 blocks/s,  40.635 Mb/s)
Attachment 38: Screenshot_from_2024-06-15_18-52-04.png
Screenshot_from_2024-06-15_18-52-04.png
Attachment 39: Screenshot_from_2024-06-15_18-52-35.png
Screenshot_from_2024-06-15_18-52-35.png
Attachment 40: Screenshot_from_2024-06-15_18-53-05.png
Screenshot_from_2024-06-15_18-53-05.png
Attachment 41: Screenshot_from_2024-06-15_18-53-49.png
Screenshot_from_2024-06-15_18-53-49.png
Attachment 42: Screenshot_from_2024-06-15_22-50-22.png
Screenshot_from_2024-06-15_22-50-22.png
Attachment 43: Screenshot_from_2024-06-15_22-50-03.png
Screenshot_from_2024-06-15_22-50-03.png
Attachment 44: Screenshot_from_2024-06-15_22-50-49.png
Screenshot_from_2024-06-15_22-50-49.png
Attachment 45: Screenshot_from_2024-06-15_22-51-18.png
Screenshot_from_2024-06-15_22-51-18.png
  663   Fri Jun 14 15:00:47 2024 Norah , Muneerah 16:00-20:00 Friday 14 June

Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00

DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)

Another set of screenshots was taken around 18:00, and everything looks good.

DSSSD bias & leakage current  ok - attachment # 5-6
FEE64 temperatures  ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11

At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.

I noted that the  bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
 it does not affect AIDA

At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA

At 22 the cooling system fixed and Nic and the other person on site restarted AIDA

 

 

 

 

 

Attachment 1: Screenshot_from_2024-06-14_16-06-09.png
Screenshot_from_2024-06-14_16-06-09.png
Attachment 2: Screenshot_from_2024-06-14_16-04-58.png
Screenshot_from_2024-06-14_16-04-58.png
Attachment 3: Screenshot_from_2024-06-14_16-04-23.png
Screenshot_from_2024-06-14_16-04-23.png
Attachment 4: Screenshot_from_2024-06-14_16-05-33.png
Screenshot_from_2024-06-14_16-05-33.png
Attachment 5: Screenshot_from_2024-06-14_18-02-36.png
Screenshot_from_2024-06-14_18-02-36.png
Attachment 6: Screenshot_from_2024-06-14_18-02-08.png
Screenshot_from_2024-06-14_18-02-08.png
Attachment 7: Screenshot_from_2024-06-14_18-04-40.png
Screenshot_from_2024-06-14_18-04-40.png
Attachment 8: Screenshot_from_2024-06-14_18-05-44.png
Screenshot_from_2024-06-14_18-05-44.png
Attachment 9: Screenshot_from_2024-06-14_18-08-50.png
Screenshot_from_2024-06-14_18-08-50.png
Attachment 10: Screenshot_from_2024-06-14_18-11-04.png
Screenshot_from_2024-06-14_18-11-04.png
Attachment 11: Screenshot_from_2024-06-14_18-17-15.png
Screenshot_from_2024-06-14_18-17-15.png
Attachment 12: Screenshot_from_2024-06-14_19-23-32.png
Screenshot_from_2024-06-14_19-23-32.png
Attachment 13: Screenshot_from_2024-06-14_19-35-11.png
Screenshot_from_2024-06-14_19-35-11.png
  662   Fri Jun 14 06:22:39 2024 TD08:00-16:00 Friday 14 June
07.23 All histograms & stats zero'd

      DAQ continues data file S181/R8_35

      analysis data file R8_35 - attachment 1
      max deadtime 4.8% (aida08), all other FEE64s < 3%
      LEC data rate 1.677M HEC data rate 3.7k

08.46 DSSSD bias & leakage current OK - attachments 2-3

      FEE64 temps OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      per p+n FEE64 1.8.L spectra - attachment 8
       aida09 pulser peak width 65 ch FWHM

      per FEE64 1.8.H spectra - attachment 7 & 10

      per FEE64 1.8.W spectra - 20us FSR - attachment 9 & 11

      Merger, TapeServer etc - attachments 12-13
       data file S181/R8_72


10.51 DSSSD bias & leakage current OK - attachments 14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Rate spectra - attachment 17

10.54 analysis data file R8_122 - attachment 18
       max deadtime 7.8% (aida08), all other FEE64s < 3%
       ADC data rate 1.88M, HEC data rate 4.7k


14.24 DSSSD bias & leakage current OK - attachments 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Rate spectra - attachment 21

      aida02 no data

14.28 merger restarted and DAQ running OK 


14.38 analysis data file R8_213 - attachment 22
       max deadtime 6.6% (aida08), all other FEE64s < 3.5%
       ADC data rate 1.935M, HEC data rate 3.6k
Attachment 1: R8_35
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257896837 ( 1677094.0 Hz)
        Other data format:    4023165 (   26162.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        943 (       6.1 Hz)
                               RESUME:        943 (       6.1 Hz)
                              SYNC100:      32573 (     211.8 Hz)
                              WR48-63:      32573 (     211.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3956133 (   25726.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     571720 (    3717.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.776 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.108                0.000
  1                0.164                0.000
  2                0.270                0.000
  3                0.595                0.000
  4                0.126                0.000
  5                1.001                0.000
  6                0.058                0.000
  7                7.436                0.000
  8                0.069                0.000
  9                3.830                0.000
 10                2.186                0.000
 11                0.125                0.000
 12                0.069                0.000
 13                0.757                0.000
 14                0.154                0.000
 15                2.216                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12981434       7806          0          0         18         18       1578       1578          0       4614          0      11424
  1    8323345       2124          0          0         12         12       1050       1050          0          0          0      25906
  2   13584435    1382069          0          0         11         11       1829       1829          0    1378389          0      15197
  3   18274735       4850          0          0         51         51       2374       2374          0          0          0      20257
  4    5283424     405074          0          0          6          6        727        727          0     403608          0       6463
  5   17252022       4608          0          0         99         99       2205       2205          0          0          0      23562
  6    5456624     322759          0          0          2          2        750        750          0     321255          0      15002
  7   48918628      12422          0          0        216        216       5995       5995          0          0          0     358328
  8    6232233       1470          0          0          2          2        733        733          0          0          0      17364
  9   40119718    1354831          0          0        148        148       5161       5161          0    1344213          0      16391
 10   18973819       5022          0          0        199        199       2312       2312          0          0          0       9722
 11    3858542     505136          0          0          4          4        537        537          0     504054          0      15811
 12    4193350        998          0          0          1          1        498        498          0          0          0       4061
 13   13770630       3644          0          0         62         62       1760       1760          0          0          0      14623
 14    4850419       1244          0          0          2          2        620        620          0          0          0      10415
 15   35823479       9108          0          0        110        110       4444       4444          0          0          0       7194
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.436s ( 1017.956 blocks/s,  63.622 Mb/s)
Attachment 2: Screenshot_from_2024-06-14_08-46-40.png
Screenshot_from_2024-06-14_08-46-40.png
Attachment 3: Screenshot_from_2024-06-14_08-47-06.png
Screenshot_from_2024-06-14_08-47-06.png
Attachment 4: Screenshot_from_2024-06-14_08-49-23.png
Screenshot_from_2024-06-14_08-49-23.png
Attachment 5: Screenshot_from_2024-06-14_08-49-56.png
Screenshot_from_2024-06-14_08-49-56.png
Attachment 6: Screenshot_from_2024-06-14_08-50-17.png
Screenshot_from_2024-06-14_08-50-17.png
Attachment 7: Screenshot_from_2024-06-14_08-53-11.png
Screenshot_from_2024-06-14_08-53-11.png
Attachment 8: Screenshot_from_2024-06-14_08-51-44.png
Screenshot_from_2024-06-14_08-51-44.png
Attachment 9: Screenshot_from_2024-06-14_08-54-47.png
Screenshot_from_2024-06-14_08-54-47.png
Attachment 10: Screenshot_from_2024-06-14_08-53-58.png
Screenshot_from_2024-06-14_08-53-58.png
Attachment 11: Screenshot_from_2024-06-14_08-55-50.png
Screenshot_from_2024-06-14_08-55-50.png
Attachment 12: Screenshot_from_2024-06-14_08-56-36.png
Screenshot_from_2024-06-14_08-56-36.png
Attachment 13: Screenshot_from_2024-06-14_08-56-13.png
Screenshot_from_2024-06-14_08-56-13.png
Attachment 14: Screenshot_from_2024-06-14_10-51-34.png
Screenshot_from_2024-06-14_10-51-34.png
Attachment 15: Screenshot_from_2024-06-14_10-53-46.png
Screenshot_from_2024-06-14_10-53-46.png
Attachment 16: Screenshot_from_2024-06-14_10-54-27.png
Screenshot_from_2024-06-14_10-54-27.png
Attachment 17: Screenshot_from_2024-06-14_10-55-24.png
Screenshot_from_2024-06-14_10-55-24.png
Attachment 18: R8_122
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257243299 ( 1885175.7 Hz)
        Other data format:    4676701 (   34272.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1054 (       7.7 Hz)
                               RESUME:       1054 (       7.7 Hz)
                              SYNC100:      32509 (     238.2 Hz)
                              WR48-63:      32509 (     238.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4609575 (   33780.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     643720 (    4717.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      136.456 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.291                0.000
  1                0.331                0.000
  2                0.072                0.000
  3                0.583                0.000
  4                0.004                0.000
  5                1.153                0.000
  6                0.002                0.000
  7               10.599                0.000
  8                0.009                0.000
  9                3.557                0.000
 10                2.108                0.000
 11                0.007                0.000
 12                0.000                0.000
 13                0.583                0.000
 14                0.004                0.000
 15                1.925                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13787962       7660          0          0         31         31       1756       1756          0       4086          0      17199
  1   11491912       2978          0          0         24         24       1465       1465          0          0          0      38699
  2   14079661    1398532          0          0         10         10       1939       1939          0    1394634          0      23119
  3   19048179       4880          0          0         43         43       2397       2397          0          0          0      31229
  4    5716791     616800          0          0          1          1        782        782          0     615234          0      10346
  5   18363851       4878          0          0        113        113       2326       2326          0          0          0      35150
  6    5742131     471711          0          0          1          1        755        755          0     470199          0      22729
  7   44929108      11550          0          0        247        247       5528       5528          0          0          0     318806
  8    6563442       1628          0          0          4          4        810        810          0          0          0      26254
  9   37095189    1366938          0          0        146        146       4783       4783          0    1357080          0      24684
 10   18370423       4942          0          0        241        241       2230       2230          0          0          0      15154
 11    4582984     769694          0          0          2          2        674        674          0     768342          0      24358
 12    4619008       1036          0          0          0          0        518        518          0          0          0       6521
 13   14138411       3650          0          0         78         78       1747       1747          0          0          0      22204
 14    5321169       1274          0          0          2          2        635        635          0          0          0      16080
 15   33393078       8550          0          0        111        111       4164       4164          0          0          0      11188
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.508s ( 1015.621 blocks/s,  63.476 Mb/s)
Attachment 19: Screenshot_from_2024-06-14_14-23-54.png
Screenshot_from_2024-06-14_14-23-54.png
Attachment 20: Screenshot_from_2024-06-14_14-24-24.png
Screenshot_from_2024-06-14_14-24-24.png
Attachment 21: Screenshot_from_2024-06-14_14-25-10.png
Screenshot_from_2024-06-14_14-25-10.png
Attachment 22: Screenshot_from_2024-06-14_14-25-38.png
Screenshot_from_2024-06-14_14-25-38.png
Attachment 23: R8_213
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258517055 ( 1934722.8 Hz)
        Other data format:    3402945 (   25467.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1143 (       8.6 Hz)
                               RESUME:       1142 (       8.5 Hz)
                              SYNC100:      32498 (     243.2 Hz)
                              WR48-63:      32498 (     243.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3335664 (   24963.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     475836 (    3561.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      133.620 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.550                0.000
  1                0.000                0.000
  2                0.143                0.000
  3                1.051                0.000
  4                0.044                0.000
  5                1.439                0.000
  6                0.003                0.000
  7                8.857                0.000
  8                0.012                0.000
  9                4.685                0.000
 10                3.463                0.000
 11                0.020                0.000
 12                0.000                0.000
 13                0.869                0.000
 14                0.008                0.000
 15                2.432                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18250814       8751          0          0         55         55       2330       2330          0       3981          0       9142
  1          0          0          0          0          0          0          0          0          0          0          0          0
  2   17501975    1182466          0          0         15         15       2393       2393          0    1177650          0      12677
  3   17523014       4524          0          0         49         49       2213       2213          0          0          0      16471
  4    6950058     338741          0          0          8          8        923        923          0     336879          0       5558
  5   16698357       4354          0          0        104        104       2073       2073          0          0          0      18504
  6    5487299     262839          0          0          1          1        716        716          0     261405          0      12626
  7   43452557      11094          0          0        210        210       5337       5337          0          0          0     321021
  8    5704212       1552          0          0          1          1        775        775          0          0          0      14649
  9   38744177    1147432          0          0        173        172       4922       4922          0    1137243          0      13757
 10   21009531       5724          0          0        299        299       2563       2563          0          0          0       7991
 11    4841879     419852          0          0          4          4        669        669          0     418506          0      13290
 12    4136803       1032          0          0          0          0        516        516          0          0          0       3489
 13   16727954       4328          0          0        114        114       2050       2050          0          0          0      12047
 14    5878222       1412          0          0          1          1        705        705          0          0          0       8774
 15   35610203       8844          0          0        109        109       4313       4313          0          0          0       5840
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.188s (  994.175 blocks/s,  62.136 Mb/s)
  661   Thu Jun 13 23:04:01 2024 Dan Judson, TD0.00-8.00 14/6/24

0.00 checks - all looks ok

Voltage/currents - attachment 1

Rates attachment 2 + 3

Temps - attachment 4

Merger - attachment 5

ucesb - attachment 6

Grafana - attachment 7

 

1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.

Datafile R7_961 being written when stopped

1.40 - seems ok

 

1.50 checks - seems ok

Voltage/currents - attachment 8

Rates attachment 9 + 10

Temps - attachment 11

Merger - attachment 12

ucesb - attachment 13

Grafana - attachment 14

 

3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1010 being written when stopped

3.38 - seems ok

 

3.40 checks - all looks ok

Voltage/currents - attachment 15

Rates attachment 16 + 17

Temps - attachment 18

Merger - attachment 19

ucesb - attachment 20

Grafana - attachment 21

 

Zeroed spectra - attachments 22-28

 

 

5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1062 being written when stopped

AIDA out of the main data

Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in 

6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it

Started data file R8

AIDA back in the main data

 

FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  17D8C4ED 8F46CF7B , WR/10=>   2627A17C18714BF, Readout Time =>   2627A17C2A74000
aida02 : White Rabbit=>  17D8C4ED A0A1B623 , WR/10=>   2627A17C3435F03, Readout Time =>   2627A17C4410000
aida03 : White Rabbit=>  17D8C4ED AFDACE68 , WR/10=>   2627A17C4C914A4, Readout Time =>   2627A17C5D3C000
aida04 : White Rabbit=>  17D8C4ED C0516FC2 , WR/10=>   2627A17C66E8B2D, Readout Time =>   2627A17C7AC4000
aida05 : White Rabbit=>  17D8C4ED D3B93DAB , WR/10=>   2627A17C85F52F7, Readout Time =>   2627A17C9AEC000
aida06 : White Rabbit=>  17D8C4ED E8C9B243 , WR/10=>   2627A17CA7A91D3, Readout Time =>   2627A17CB704000
aida07 : White Rabbit=>  17D8C4ED FFE475FD , WR/10=>   2627A17CCCA0BCC, Readout Time =>   2627A17CDD94000
aida08 : White Rabbit=>  17D8C4EE 103232A6 , WR/10=>   2627A17CE6B6B77, Readout Time =>   2627A17CF7BC000
aida09 : White Rabbit=>  17D8C4EE 20A1F612 , WR/10=>   2627A17D0103235, Readout Time =>   2627A17D112C000
aida10 : White Rabbit=>  17D8C4EE 307D8FE7 , WR/10=>   2627A17D1A627FD, Readout Time =>   2627A17D2958000
aida11 : White Rabbit=>  17D8C4EE 4402829D , WR/10=>   2627A17D399D9DC, Readout Time =>   2627A17D4570000
aida12 : White Rabbit=>  17D8C4EE 6288B04D , WR/10=>   2627A17D6A744D4, Readout Time =>   2627A17D7D94000
aida13 : White Rabbit=>  17D8C4EE 751E1306 , WR/10=>   2627A17D88301E7, Readout Time =>   2627A17D9C34000
aida14 : White Rabbit=>  17D8C4EE 894E8806 , WR/10=>   2627A17DA87DA67, Readout Time =>   2627A17DBFF4000
aida15 : White Rabbit=>  17D8C4EE 9EEEB997 , WR/10=>   2627A17DCB178F5, Readout Time =>   2627A17DDC18000
aida16 : White Rabbit=>  17D8C4EE AF0BC923 , WR/10=>   2627A17DE4DFA83, Readout Time =>   2627A17DF5C8000

Tom thinks everything appears ok with AIDA

Have converted WR timestamp to data/time - see attachment 40

At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale.. 

However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.

 

 

7.00 checks - all looks ok

Voltage/currents - attachment 31

Rates attachment 32 + 33

Temps - attachment 36

Merger - attachment 37

ucesb - attachment 39

Grafana - attachment 38

 

Handed over to Tom

 

Attachment 1: Screenshot_from_2024-06-13_23-58-44.png
Screenshot_from_2024-06-13_23-58-44.png
Attachment 2: Screenshot_from_2024-06-13_23-59-34.png
Screenshot_from_2024-06-13_23-59-34.png
Attachment 3: Screenshot_from_2024-06-14_00-00-09.png
Screenshot_from_2024-06-14_00-00-09.png
Attachment 4: Screenshot_from_2024-06-14_00-00-43.png
Screenshot_from_2024-06-14_00-00-43.png
Attachment 5: Screenshot_from_2024-06-14_00-01-36.png
Screenshot_from_2024-06-14_00-01-36.png
Attachment 6: Screenshot_from_2024-06-14_00-02-45.png
Screenshot_from_2024-06-14_00-02-45.png
Attachment 7: Screenshot_from_2024-06-14_00-03-26.png
Screenshot_from_2024-06-14_00-03-26.png
Attachment 8: Screenshot_from_2024-06-14_01-47-38.png
Screenshot_from_2024-06-14_01-47-38.png
Attachment 9: Screenshot_from_2024-06-14_01-48-12.png
Screenshot_from_2024-06-14_01-48-12.png
Attachment 10: Screenshot_from_2024-06-14_01-48-45.png
Screenshot_from_2024-06-14_01-48-45.png
Attachment 11: Screenshot_from_2024-06-14_01-49-16.png
Screenshot_from_2024-06-14_01-49-16.png
Attachment 12: Screenshot_from_2024-06-14_01-49-48.png
Screenshot_from_2024-06-14_01-49-48.png
Attachment 13: Screenshot_from_2024-06-14_01-50-46.png
Screenshot_from_2024-06-14_01-50-46.png
Attachment 14: Screenshot_from_2024-06-14_01-51-18.png
Screenshot_from_2024-06-14_01-51-18.png
Attachment 15: Screenshot_from_2024-06-14_03-40-33.png
Screenshot_from_2024-06-14_03-40-33.png
Attachment 16: Screenshot_from_2024-06-14_03-40-56.png
Screenshot_from_2024-06-14_03-40-56.png
Attachment 17: Screenshot_from_2024-06-14_03-41-35.png
Screenshot_from_2024-06-14_03-41-35.png
Attachment 18: Screenshot_from_2024-06-14_03-42-15.png
Screenshot_from_2024-06-14_03-42-15.png
Attachment 19: Screenshot_from_2024-06-14_03-42-42.png
Screenshot_from_2024-06-14_03-42-42.png
Attachment 20: Screenshot_from_2024-06-14_03-43-39.png
Screenshot_from_2024-06-14_03-43-39.png
Attachment 21: Screenshot_from_2024-06-14_03-44-11.png
Screenshot_from_2024-06-14_03-44-11.png
Attachment 22: Screenshot_from_2024-06-14_03-53-08.png
Screenshot_from_2024-06-14_03-53-08.png
Attachment 23: Screenshot_from_2024-06-14_03-53-51.png
Screenshot_from_2024-06-14_03-53-51.png
Attachment 24: Screenshot_from_2024-06-14_03-54-36.png
Screenshot_from_2024-06-14_03-54-36.png
Attachment 25: Screenshot_from_2024-06-14_03-55-19.png
Screenshot_from_2024-06-14_03-55-19.png
Attachment 26: Screenshot_from_2024-06-14_03-55-50.png
Screenshot_from_2024-06-14_03-55-50.png
Attachment 27: Screenshot_from_2024-06-14_03-56-56.png
Screenshot_from_2024-06-14_03-56-56.png
Attachment 28: Screenshot_from_2024-06-14_03-58-10.png
Screenshot_from_2024-06-14_03-58-10.png
Attachment 29: Screenshot_from_2024-06-14_05-52-47.png
Screenshot_from_2024-06-14_05-52-47.png
Attachment 30: Screenshot_2024-06-14_at_05.31.21.png
Screenshot_2024-06-14_at_05.31.21.png
Attachment 31: Screenshot_from_2024-06-14_07-02-14.png
Screenshot_from_2024-06-14_07-02-14.png
Attachment 32: Screenshot_from_2024-06-14_07-00-53.png
Screenshot_from_2024-06-14_07-00-53.png
Attachment 33: Screenshot_from_2024-06-14_07-01-20.png
Screenshot_from_2024-06-14_07-01-20.png
Attachment 34: Screenshot_from_2024-06-14_07-01-55.png
Screenshot_from_2024-06-14_07-01-55.png
Attachment 35: Screenshot_from_2024-06-14_07-02-47.png
Screenshot_from_2024-06-14_07-02-47.png
Attachment 36: Screenshot_from_2024-06-14_07-01-55.png
Screenshot_from_2024-06-14_07-01-55.png
Attachment 37: Screenshot_from_2024-06-14_07-02-47.png
Screenshot_from_2024-06-14_07-02-47.png
Attachment 38: Screenshot_from_2024-06-14_07-03-23.png
Screenshot_from_2024-06-14_07-03-23.png
Attachment 39: Screenshot_from_2024-06-14_07-04-01.png
Screenshot_from_2024-06-14_07-04-01.png
Attachment 40: Capture.PNG
Capture.PNG
  660   Thu Jun 13 21:06:44 2024 Robert Page16:00 to 00:00 shift on 13/6/2024

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

 

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

 

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

 

 

Attachment 1: Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Attachment 2: Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Attachment 6: Screenshot_from_2024-06-13_16-04-29.png
Screenshot_from_2024-06-13_16-04-29.png
Attachment 7: Screenshot_from_2024-06-13_17-34-30.png
Screenshot_from_2024-06-13_17-34-30.png
Attachment 8: Screenshot_from_2024-06-13_18-02-57.png
Screenshot_from_2024-06-13_18-02-57.png
Attachment 9: Screenshot_2024-06-13_at_18-01-10_ucesb.png
Screenshot_2024-06-13_at_18-01-10_ucesb.png
Attachment 10: Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 11: Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Attachment 13: Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Attachment 14: Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Attachment 16: Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: Screenshot_2024-06-13_at_20-57-39_ucesb.png
Screenshot_2024-06-13_at_20-57-39_ucesb.png
Attachment 18: Screenshot_from_2024-06-13_20-55-04.png
Screenshot_from_2024-06-13_20-55-04.png
Attachment 19: Screenshot_from_2024-06-13_22-52-59.png
Screenshot_from_2024-06-13_22-52-59.png
Attachment 20: Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Attachment 21: Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Attachment 22: Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 23: Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Attachment 24: Screenshot_2024-06-13_at_22-56-12_ucesb.png
Screenshot_2024-06-13_at_22-56-12_ucesb.png
Attachment 25: Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Attachment 26: Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
  659   Thu Jun 13 06:49:54 2024 Magda Satrazani08:00 - 16:00 Thursday 13 June 2024

Took over the shift from Marc.

08:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

10:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12

12:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 13-14
FEE64 temperatures  ok - attachment # 15
ADC data item stats - attachment # 16
Merger ok - Attachement # 17
Tape service - attachement # 18

 

 

12.34 analysis data file S181/R7_610 - attachment 19

max deadtime 8.7% (aida08), all other FEE64s < 2.7%

LEC data rate 2.000M, HEC data rate 4.7k

 

12.52 per p+n FEE64 1.8.L spectra - attachment 20

 aida09 pulser peak width 63 ch FWHM

per FEE64 1.8.H spectra - attachments 21-22

per FEE64 1.8.W spectra - 20us FSR - attachments 23-24

 

13.00 all histograms zero'd

 

14:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 19-20
FEE64 temperatures  ok - attachment # 21
ADC data item stats - attachment # 22
Merger ok - Attachement # 23
Tape service - attachement # 24

 

13.01 Beam OFF for S4 access

ADC data item stats - attachment 25

per FEE64 Rate spectra - attachment 26

14.12 analysis data file R7_654 - attachment 27

max deadtime 3.9% (aida08), all other FEE64s < 3%

LEC data rate 1.817M, HEC data rate 2.5k

14.00 data file R7_649 - attachment 28

max deadtime 4.5% (aida08), all other FEE64s < 3%

LEC data rate 1.756M, HEC data rate 2.5k

 

 

16:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 25-26
FEE64 temperatures  ok - attachment # 27
ADC data item stats - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30

Attachment 1: Screenshot_2024-06-13_at_07-54-16_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_07-54-16_AIDA_-_Grafana.png
Attachment 2: Screenshot_from_2024-06-13_07-52-18.png
Screenshot_from_2024-06-13_07-52-18.png
Attachment 3: Screenshot_2024-06-13_at_07-56-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_07-56-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2024-06-13_at_07-58-19_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_07-58-19_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2024-06-13_at_08-02-24_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_08-02-24_NewMerger_Control_aidas-gsi.png
Attachment 6: Screenshot_2024-06-13_at_08-03-42_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_08-03-42_Tape_Service_(Expert)_aidas-gsi.png
Attachment 7: Screenshot_2024-06-13_at_10-03-46_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_10-03-46_AIDA_-_Grafana.png
Attachment 8: Screenshot_from_2024-06-13_10-05-11.png
Screenshot_from_2024-06-13_10-05-11.png
Attachment 9: Screenshot_2024-06-13_at_10-07-46_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_10-07-46_Temperature_and_status_scan_aidas-gsi.png
Attachment 10: Screenshot_2024-06-13_at_10-08-20_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_10-08-20_Statistics_aidas-gsi.png
Attachment 11: Screenshot_2024-06-13_at_10-10-56_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_10-10-56_NewMerger_Control_aidas-gsi.png
Attachment 12: Screenshot_2024-06-13_at_10-11-14_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_10-11-14_Tape_Service_(Expert)_aidas-gsi.png
Attachment 13: Screenshot_2024-06-13_at_12-03-03_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_12-03-03_AIDA_-_Grafana.png
Attachment 14: Screenshot_from_2024-06-13_12-03-51.png
Screenshot_from_2024-06-13_12-03-51.png
Attachment 15: Screenshot_2024-06-13_at_12-04-46_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_12-04-46_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_2024-06-13_at_12-05-04_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_12-05-04_Statistics_aidas-gsi.png
Attachment 17: Screenshot_2024-06-13_at_12-06-12_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_12-06-12_NewMerger_Control_aidas-gsi.png
Attachment 18: Screenshot_2024-06-13_at_12-06-30_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_12-06-30_Tape_Service_(Expert)_aidas-gsi.png
Attachment 19: R7_610
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257594733 ( 2000279.2 Hz)
        Other data format:    4325267 (   33586.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1104 (       8.6 Hz)
                               RESUME:       1104 (       8.6 Hz)
                              SYNC100:      32479 (     252.2 Hz)
                              WR48-63:      32479 (     252.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4258101 (   33065.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     602185 (    4676.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      128.779 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.378                0.000
  1                0.764                0.000
  2                0.139                0.000
  3                0.656                0.000
  4                0.011                0.000
  5                1.499                0.000
  6                0.000                0.000
  7               11.233                0.000
  8                0.013                0.000
  9                3.452                0.000
 10                2.528                0.000
 11                0.011                0.000
 12                0.000                0.000
 13                0.520                0.000
 14                0.005                0.000
 15                2.249                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15133196       7799          0          0         47         47       1922       1922          0       3861          0      15842
  1   11745456       3032          0          0         27         27       1489       1489          0          0          0      35232
  2   14852161    1309647          0          0         18         18       1989       1989          0    1305633          0      20871
  3   18562801       4858          0          0         50         50       2379       2379          0          0          0      27762
  4    6102528     560384          0          0          2          2        773        773          0     558834          0       9246
  5   18133649       4756          0          0        124        124       2254       2254          0          0          0      32497
  6    5708332     425318          0          0          0          0        727        727          0     423864          0      20696
  7   42953157      11164          0          0        244        244       5338       5338          0          0          0     306073
  8    6253278       1506          0          0          2          2        751        751          0          0          0      24189
  9   36712642    1280462          0          0        132        132       4801       4801          0    1270596          0      22769
 10   18971051       5070          0          0        273        273       2262       2262          0          0          0      13720
 11    4672661     696545          0          0          2          2        614        614          0     695313          0      22243
 12    4500765       1148          0          0          1          1        573        573          0          0          0       5817
 13   14826894       3760          0          0         66         66       1814       1814          0          0          0      20573
 14    5503214       1378          0          0          1          1        688        688          0          0          0      14645
 15   32962948       8440          0          0        115        115       4105       4105          0          0          0      10010
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.625s (  717.087 blocks/s,  44.818 Mb/s)
Attachment 20: Screenshot_from_2024-06-13_12-52-29.png
Screenshot_from_2024-06-13_12-52-29.png
Attachment 21: Screenshot_from_2024-06-13_12-54-14.png
Screenshot_from_2024-06-13_12-54-14.png
Attachment 22: Screenshot_from_2024-06-13_12-53-37.png
Screenshot_from_2024-06-13_12-53-37.png
Attachment 23: Screenshot_from_2024-06-13_12-55-33.png
Screenshot_from_2024-06-13_12-55-33.png
Attachment 24: Screenshot_from_2024-06-13_12-54-56.png
Screenshot_from_2024-06-13_12-54-56.png
Attachment 25: Screenshot_2024-06-13_at_14-03-30_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_14-03-30_AIDA_-_Grafana.png
Attachment 26: Screenshot_from_2024-06-13_14-03-07.png
Screenshot_from_2024-06-13_14-03-07.png
Attachment 27: Screenshot_2024-06-13_at_14-05-01_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_14-05-01_Temperature_and_status_scan_aidas-gsi.png
Attachment 28: Screenshot_2024-06-13_at_14-05-24_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_14-05-24_Statistics_aidas-gsi.png
Attachment 29: Screenshot_2024-06-13_at_14-06-51_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_14-06-51_NewMerger_Control_aidas-gsi.png
Attachment 30: Screenshot_2024-06-13_at_14-07-11_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_14-07-11_Tape_Service_(Expert)_aidas-gsi.png
Attachment 31: Screenshot_from_2024-06-13_14-17-01.png
Screenshot_from_2024-06-13_14-17-01.png
Attachment 32: Screenshot_from_2024-06-13_14-19-05.png
Screenshot_from_2024-06-13_14-19-05.png
Attachment 33: R7_654
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259951011 ( 1816478.8 Hz)
        Other data format:    1968989 (   13758.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        987 (       6.9 Hz)
                               RESUME:        987 (       6.9 Hz)
                              SYNC100:      32533 (     227.3 Hz)
                              WR48-63:      32533 (     227.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1901949 (   13290.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354246 (    2475.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      143.107 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.509                0.000
  1                0.000                0.000
  2                0.189                0.000
  3                0.238                0.000
  4                0.004                0.000
  5                0.314                0.000
  6                0.000                0.000
  7                5.616                0.000
  8                0.000                0.000
  9                4.141                0.000
 10                3.021                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.437                0.000
 14                0.000                0.000
 15                2.204                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17180431       8836          0          0         56         56       2220       2220          0       4284          0          0
  1    3455015        914          0          0          0          0        457        457          0          0          0          0
  2   16605423     962341          0          0         24         24       2144       2144          0     958005          0          0
  3   14977123       3970          0          0         39         39       1946       1946          0          0          0          0
  4    6343544       1554          0          0          1          1        776        776          0          0          0          0
  5   14745967       3758          0          0         68         68       1811       1811          0          0          0          0
  6    4885840       9840          0          0          0          0        627        627          0       8586          0          0
  7   46653553      11994          0          0        167        167       5830       5830          0          0          0     354246
  8    5022167       1302          0          0          0          0        651        651          0          0          0          0
  9   41378629     941868          0          0        154        154       5243       5243          0     931074          0          0
 10   21774767       5944          0          0        304        304       2668       2668          0          0          0          0
 11    3829761       1030          0          0          0          0        515        515          0          0          0          0
 12    3406542        774          0          0          0          0        387        387          0          0          0          0
 13   16951477       4402          0          0         63         63       2138       2138          0          0          0          0
 14    5065059       1154          0          0          0          0        577        577          0          0          0          0
 15   37675713       9308          0          0        111        111       4543       4543          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.719s (  767.041 blocks/s,  47.940 Mb/s)
Attachment 34: R7_649
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259542903 ( 1756063.7 Hz)
        Other data format:    2377097 (   16083.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        995 (       6.7 Hz)
                               RESUME:        994 (       6.7 Hz)
                              SYNC100:      32551 (     220.2 Hz)
                              WR48-63:      32551 (     220.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2310006 (   15629.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     373724 (    2528.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      147.798 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.337                0.000
  1                0.072                0.000
  2                0.087                0.000
  3                0.196                0.000
  4                0.011                0.000
  5                0.401                0.000
  6                0.000                0.000
  7                6.652                0.000
  8                0.000                0.000
  9                3.901                0.000
 10                2.506                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.447                0.000
 14                0.000                0.000
 15                2.451                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16241246       8589          0          0         45         45       2034       2034          0       4431          0       1142
  1    4109651       1104          0          0          4          4        548        548          0          0          0       2470
  2   15921172    1109604          0          0         15         15       2046       2046          0    1105482          0       1509
  3   15655639       3862          0          0         30         30       1901       1901          0          0          0       1938
  4    5992464      40675          0          0          2          2        792        792          0      39087          0        732
  5   15116945       3940          0          0         57         57       1913       1913          0          0          0       2309
  6    4949702      39054          0          0          0          0        642        642          0      37770          0       1486
  7   47384271      12428          0          0        197        197       6017       6017          0          0          0     352995
  8    5162177       1276          0          0          0          0        638        638          0          0          0       1588
  9   41669335    1085810          0          0        164        163       5344       5344          0    1074795          0       1446
 10   21487589       5832          0          0        286        286       2630       2630          0          0          0        957
 11    3666376      49337          0          0          1          1        447        447          0      48441          0       1546
 12    3498670        892          0          0          0          0        446        446          0          0          0        448
 13   16139721       3982          0          0         64         64       1927       1927          0          0          0       1446
 14    4880733       1226          0          0          0          0        613        613          0          0          0       1032
 15   37667212       9486          0          0        130        130       4613       4613          0          0          0        680
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   46.664s (  685.753 blocks/s,  42.860 Mb/s)
Attachment 35: Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Attachment 36: Screenshot_from_2024-06-13_16-04-29.png
Screenshot_from_2024-06-13_16-04-29.png
Attachment 37: Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 38: Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Attachment 39: Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Attachment 40: Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
  658   Wed Jun 12 23:00:16 2024 Marc00:00-08:00 Thursday 13 June

Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

 

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

 

Attachment 1: Screenshot_from_2024-06-13_01-04-30.png
Screenshot_from_2024-06-13_01-04-30.png
Attachment 2: Screenshot_from_2024-06-13_01-06-23.png
Screenshot_from_2024-06-13_01-06-23.png
Attachment 3: Screenshot_from_2024-06-13_01-07-02.png
Screenshot_from_2024-06-13_01-07-02.png
Attachment 4: Screenshot_from_2024-06-13_01-07-36.png
Screenshot_from_2024-06-13_01-07-36.png
Attachment 5: Screenshot_from_2024-06-13_01-08-39.png
Screenshot_from_2024-06-13_01-08-39.png
Attachment 6: Screenshot_from_2024-06-13_01-08-02.png
Screenshot_from_2024-06-13_01-08-02.png
Attachment 7: Screenshot_from_2024-06-13_04-41-04.png
Screenshot_from_2024-06-13_04-41-04.png
Attachment 8: Screenshot_from_2024-06-13_04-41-38.png
Screenshot_from_2024-06-13_04-41-38.png
Attachment 9: Screenshot_from_2024-06-13_04-42-19.png
Screenshot_from_2024-06-13_04-42-19.png
Attachment 10: Screenshot_from_2024-06-13_04-42-45.png
Screenshot_from_2024-06-13_04-42-45.png
Attachment 11: Screenshot_from_2024-06-13_04-46-50.png
Screenshot_from_2024-06-13_04-46-50.png
Attachment 12: Screenshot_from_2024-06-13_04-47-09.png
Screenshot_from_2024-06-13_04-47-09.png
Attachment 13: Screenshot_from_2024-06-13_04-45-02.png
Screenshot_from_2024-06-13_04-45-02.png
Attachment 14: Screenshot_from_2024-06-13_04-48-07.png
Screenshot_from_2024-06-13_04-48-07.png
Attachment 15: Screenshot_from_2024-06-13_06-01-55.png
Screenshot_from_2024-06-13_06-01-55.png
Attachment 16: Screenshot_from_2024-06-13_06-02-13.png
Screenshot_from_2024-06-13_06-02-13.png
Attachment 17: Screenshot_from_2024-06-13_06-03-06.png
Screenshot_from_2024-06-13_06-03-06.png
Attachment 18: Screenshot_from_2024-06-13_06-02-41.png
Screenshot_from_2024-06-13_06-02-41.png
Attachment 19: Screenshot_from_2024-06-13_06-04-13.png
Screenshot_from_2024-06-13_06-04-13.png
Attachment 20: Screenshot_from_2024-06-13_06-04-37.png
Screenshot_from_2024-06-13_06-04-37.png
Attachment 21: Screenshot_from_2024-06-13_06-03-37.png
Screenshot_from_2024-06-13_06-03-37.png
Attachment 22: Screenshot_from_2024-06-13_06-05-30.png
Screenshot_from_2024-06-13_06-05-30.png
Attachment 23: R7_457
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257532265 ( 1788132.4 Hz)
        Other data format:    4387735 (   30465.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1032 (       7.2 Hz)
                               RESUME:       1032 (       7.2 Hz)
                              SYNC100:      32537 (     225.9 Hz)
                              WR48-63:      32537 (     225.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4320597 (   29999.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     613796 (    4261.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.023 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.236                0.000
  1                0.214                0.000
  2                0.096                0.000
  3                0.448                0.000
  4                0.012                0.000
  5                0.887                0.000
  6                0.003                0.000
  7                8.383                0.000
  8                0.002                0.000
  9                3.230                0.000
 10                1.587                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.540                0.000
 14                0.000                0.000
 15                1.978                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13645957       7863          0          0         38         38       1738       1738          0       4311          0      14717
  1    9668775       2474          0          0         22         22       1215       1215          0          0          0      34051
  2   13971257    1386114          0          0         14         14       1900       1900          0    1382286          0      19760
  3   18385419       4726          0          0         51         51       2312       2312          0          0          0      26170
  4    5575749     526316          0          0          3          3        775        775          0     524760          0       8695
  5   17895997       4760          0          0        110        110       2270       2270          0          0          0      31206
  6    5682061     408056          0          0          1          1        783        783          0     406488          0      19551
  7   46828757      11780          0          0        204        204       5686       5686          0          0          0     333264
  8    6455085       1532          0          0          1          1        765        765          0          0          0      22949
  9   38428621    1359707          0          0        141        141       4975       4975          0    1349475          0      21660
 10   18781575       5148          0          0        247        247       2327       2327          0          0          0      13075
 11    4229184     654561          0          0          1          1        641        641          0     653277          0      20787
 12    4432128       1092          0          0          0          0        546        546          0          0          0       5445
 13   14014813       3450          0          0         86         86       1639       1639          0          0          0      19095
 14    5156679       1244          0          0          0          0        622        622          0          0          0      13849
 15   34380208       8912          0          0        113        113       4343       4343          0          0          0       9522
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   52.717s (  607.017 blocks/s,  37.939 Mb/s)
  657   Wed Jun 12 15:41:07 2024 Betool Alayed16:00-00:00 Wed 12 Jun 24

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

Attachment 1: Screenshot_from_2024-06-12_17-00-31.png
Screenshot_from_2024-06-12_17-00-31.png
Attachment 2: Screenshot_from_2024-06-12_17-01-10.png
Screenshot_from_2024-06-12_17-01-10.png
Attachment 3: Screenshot_from_2024-06-12_17-02-29.png
Screenshot_from_2024-06-12_17-02-29.png
Attachment 4: Screenshot_from_2024-06-12_17-03-14.png
Screenshot_from_2024-06-12_17-03-14.png
Attachment 5: Screenshot_from_2024-06-12_17-31-47.png
Screenshot_from_2024-06-12_17-31-47.png
Attachment 6: Screenshot_from_2024-06-12_17-31-13.png
Screenshot_from_2024-06-12_17-31-13.png
Attachment 7: Screenshot_from_2024-06-12_19-03-51.png
Screenshot_from_2024-06-12_19-03-51.png
Attachment 8: Screenshot_from_2024-06-12_19-04-39.png
Screenshot_from_2024-06-12_19-04-39.png
Attachment 9: Screenshot_from_2024-06-12_19-05-34.png
Screenshot_from_2024-06-12_19-05-34.png
Attachment 10: Screenshot_from_2024-06-12_19-06-05.png
Screenshot_from_2024-06-12_19-06-05.png
Attachment 11: Screenshot_from_2024-06-12_19-07-15.png
Screenshot_from_2024-06-12_19-07-15.png
Attachment 12: Screenshot_from_2024-06-12_19-08-02.png
Screenshot_from_2024-06-12_19-08-02.png
Attachment 13: Screenshot_from_2024-06-12_19-12-15.png
Screenshot_from_2024-06-12_19-12-15.png
Attachment 14: Screenshot_from_2024-06-12_21-01-54.png
Screenshot_from_2024-06-12_21-01-54.png
Attachment 15: Screenshot_from_2024-06-12_21-03-08.png
Screenshot_from_2024-06-12_21-03-08.png
Attachment 16: Screenshot_from_2024-06-12_21-03-52.png
Screenshot_from_2024-06-12_21-03-52.png
Attachment 17: Screenshot_from_2024-06-12_21-04-48.png
Screenshot_from_2024-06-12_21-04-48.png
Attachment 18: Screenshot_from_2024-06-12_21-06-37.png
Screenshot_from_2024-06-12_21-06-37.png
Attachment 19: Screenshot_from_2024-06-12_21-07-16.png
Screenshot_from_2024-06-12_21-07-16.png
Attachment 20: Screenshot_from_2024-06-12_21-08-36.png
Screenshot_from_2024-06-12_21-08-36.png
Attachment 21: Screenshot_from_2024-06-12_22-58-59.png
Screenshot_from_2024-06-12_22-58-59.png
Attachment 22: Screenshot_from_2024-06-12_22-59-43.png
Screenshot_from_2024-06-12_22-59-43.png
Attachment 23: Screenshot_from_2024-06-12_23-01-09.png
Screenshot_from_2024-06-12_23-01-09.png
Attachment 24: Screenshot_from_2024-06-12_23-02-14.png
Screenshot_from_2024-06-12_23-02-14.png
Attachment 25: Screenshot_from_2024-06-12_23-03-09.png
Screenshot_from_2024-06-12_23-03-09.png
Attachment 26: Screenshot_from_2024-06-12_23-03-48.png
Screenshot_from_2024-06-12_23-03-48.png
Attachment 27: Screenshot_from_2024-06-12_23-05-26.png
Screenshot_from_2024-06-12_23-05-26.png
  656   Wed Jun 12 11:06:55 2024 PPMid-shift checks, 12:00

All seems normal.

Screenshots attached.

Attachment 1: Screenshot_from_2024-06-12_12-07-32.png
Screenshot_from_2024-06-12_12-07-32.png
Attachment 2: Screenshot_from_2024-06-12_12-07-59.png
Screenshot_from_2024-06-12_12-07-59.png
Attachment 3: Screenshot_from_2024-06-12_12-08-24.png
Screenshot_from_2024-06-12_12-08-24.png
Attachment 4: Screenshot_from_2024-06-12_12-09-09.png
Screenshot_from_2024-06-12_12-09-09.png
Attachment 5: Screenshot_from_2024-06-12_12-10-02.png
Screenshot_from_2024-06-12_12-10-02.png
Attachment 6: Screenshot_from_2024-06-12_12-10-30.png
Screenshot_from_2024-06-12_12-10-30.png
Attachment 7: Screenshot_from_2024-06-12_12-10-46.png
Screenshot_from_2024-06-12_12-10-46.png
Attachment 8: Screenshot_from_2024-06-12_12-11-41.png
Screenshot_from_2024-06-12_12-11-41.png
  655   Wed Jun 12 06:17:23 2024 TD08:00-16:00 Wednesday 12 June
07:10 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      Merger, TapeServer etc - attachment 5

      per p+n FEE64 1.8.L spectra - attachment 6
       aida09 pulser peak width 68 ch FWHM 
07.13  peak area 25253 total area 27548 in 221mins => 1.9-2.1Hz => deadtime < 5% or less which is consistent with data file analysis below

      analysis data file S181/R6_274 - attachment 
       max FEE64 deadtime 24% aida08
       HEC data rate 4.5kHz, LEC data rate 2.284M


14.00 DAQ crashed by accident
      FEE64s power cycled 
      DAQ restarted
      data file S181/R7

      DSSSD bias & leakage current OK - attachments 8-9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11

      WR timestamps OK - attachment 12

      All system wide checks *except* as reported attachments 13-15

      per FEE64 Rate spectra - attachment 16

      per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 66 ch FWHM

      per FEE64 1.8.H spectra - attachments 18-19

      per FEE64 1.8.W spectra - 20us FSR - attachment 20-21

      Merger, TapeServer etc - attachments 22-23
       data file S181/R7_16

      analysis data file S181/R7_16
       max deadtime 7.3% aida08, all other FEE64s < 2%
       LEC data rate 1.877M HEC data rate 4.7k
Attachment 1: Screenshot_from_2024-06-12_07-10-38.png
Screenshot_from_2024-06-12_07-10-38.png
Attachment 2: Screenshot_from_2024-06-12_07-13-55.png
Screenshot_from_2024-06-12_07-13-55.png
Attachment 3: Screenshot_from_2024-06-12_07-14-23.png
Screenshot_from_2024-06-12_07-14-23.png
Attachment 4: Screenshot_from_2024-06-12_07-15-39.png
Screenshot_from_2024-06-12_07-15-39.png
Attachment 5: Screenshot_from_2024-06-12_07-16-37.png
Screenshot_from_2024-06-12_07-16-37.png
Attachment 6: Screenshot_from_2024-06-12_07-13-22.png
Screenshot_from_2024-06-12_07-13-22.png
Attachment 7: R6_506
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258515762 ( 2284450.0 Hz)
        Other data format:    3404238 (   30082.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1713 (      15.1 Hz)
                               RESUME:       1713 (      15.1 Hz)
                              SYNC100:      32421 (     286.5 Hz)
                              WR48-63:      32421 (     286.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3335970 (   29479.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     510832 (    4514.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      113.163 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.702                0.000
  1                9.626                0.000
  2                0.301                0.000
  3               21.630                0.000
  4                2.306                0.000
  5               22.211                0.000
  6                0.011                0.000
  7               26.657                0.000
  8                0.007                0.000
  9                3.393                0.000
 10                2.227                0.000
 11                0.257                0.000
 12                1.375                0.000
 13                0.345                0.000
 14                0.006                0.000
 15                1.380                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13756770       7112          0          0         62         62       1805       1805          0       3378          0      31438
  1   16571054       4544          0          0        118        118       2154       2154          0          0          0      40517
  2   12913225    1100195          0          0         22         22       1734       1734          0    1096683          0      15005
  3   22509949       6366          0          0        388        388       2795       2795          0          0          0      25941
  4   11433492     382287          0          0         38         38       1426       1426          0     379359          0      11107
  5   23747115       6330          0          0        227        227       2938       2938          0          0          0      25861
  6    5606649     311005          0          0          2          2        708        708          0     309585          0      13358
  7   37077077       9680          0          0        293        293       4547       4547          0          0          0     194815
  8    5480756       1296          0          0          1          1        647        647          0          0          0      14388
  9   29391154    1072320          0          0        141        141       3828       3828          0    1064382          0      12206
 10   15104540       4112          0          0        210        210       1846       1846          0          0          0       7729
 11    8762807     484875          0          0         23         23       1123       1123          0     482583          0      43011
 12   10240819       2646          0          0         83         83       1240       1240          0          0          0      10084
 13   13149998       3348          0          0         39         39       1635       1635          0          0          0      22309
 14    4835687       1232          0          0          1          1        615        615          0          0          0       8794
 15   27934670       6890          0          0         65         65       3380       3380          0          0          0      34269
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.514s (  876.384 blocks/s,  54.774 Mb/s)
Attachment 8: Screenshot_from_2024-06-12_14-48-17.png
Screenshot_from_2024-06-12_14-48-17.png
Attachment 9: Screenshot_from_2024-06-12_14-48-10.png
Screenshot_from_2024-06-12_14-48-10.png
Attachment 10: Screenshot_from_2024-06-12_14-48-34.png
Screenshot_from_2024-06-12_14-48-34.png
Attachment 11: Screenshot_from_2024-06-12_14-48-44.png
Screenshot_from_2024-06-12_14-48-44.png
Attachment 12: Screenshot_from_2024-06-12_14-50-11.png
Screenshot_from_2024-06-12_14-50-11.png
Attachment 13: Screenshot_from_2024-06-12_14-49-42.png
Screenshot_from_2024-06-12_14-49-42.png
Attachment 14: Screenshot_from_2024-06-12_14-49-38.png
Screenshot_from_2024-06-12_14-49-38.png
Attachment 15: Screenshot_from_2024-06-12_14-49-03.png
Screenshot_from_2024-06-12_14-49-03.png
Attachment 16: Screenshot_from_2024-06-12_14-50-44.png
Screenshot_from_2024-06-12_14-50-44.png
Attachment 17: Screenshot_from_2024-06-12_14-52-04.png
Screenshot_from_2024-06-12_14-52-04.png
Attachment 18: Screenshot_from_2024-06-12_14-53-15.png
Screenshot_from_2024-06-12_14-53-15.png
Attachment 19: Screenshot_from_2024-06-12_14-52-48.png
Screenshot_from_2024-06-12_14-52-48.png
Attachment 20: Screenshot_from_2024-06-12_14-54-44.png
Screenshot_from_2024-06-12_14-54-44.png
Attachment 21: Screenshot_from_2024-06-12_14-54-19.png
Screenshot_from_2024-06-12_14-54-19.png
Attachment 22: Screenshot_from_2024-06-12_14-55-06.png
Screenshot_from_2024-06-12_14-55-06.png
Attachment 23: Screenshot_from_2024-06-12_14-54-57.png
Screenshot_from_2024-06-12_14-54-57.png
Attachment 24: R7_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257426271 ( 1876729.6 Hz)
        Other data format:    4493729 (   32760.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        958 (       7.0 Hz)
                               RESUME:        958 (       7.0 Hz)
                              SYNC100:      32511 (     237.0 Hz)
                              WR48-63:      32511 (     237.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4426791 (   32272.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     640590 (    4670.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.167 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.216                0.000
  1                0.956                0.000
  2                0.109                0.000
  3                1.065                0.000
  4                0.003                0.000
  5                1.328                0.000
  6                0.000                0.000
  7               10.048                0.000
  8                0.006                0.000
  9                2.496                0.000
 10                1.879                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.358                0.000
 14                0.002                0.000
 15                1.716                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13230081       7571          0          0         32         32       1703       1703          0       4101          0      17508
  1   13090516       3490          0          0         40         40       1705       1705          0          0          0      37780
  2   13947474    1365581          0          0         17         17       1952       1952          0    1361643          0      23114
  3   19918409       5228          0          0         54         54       2560       2560          0          0          0      29816
  4    5368632     580346          0          0          2          2        689        689          0     578964          0       9807
  5   18843779       4956          0          0        116        116       2362       2362          0          0          0      35331
  6    5796835     443912          0          0          1          1        735        735          0     442440          0      22936
  7   45421367      11632          0          0        240        240       5576       5576          0          0          0     316545
  8    6655009       1606          0          0          2          2        801        801          0          0          0      26785
  9   36628182    1346978          0          0        117        117       4723       4723          0    1337298          0      25590
 10   18208001       4832          0          0        188        188       2228       2228          0          0          0      15146
 11    4458699     703685          0          0          0          0        670        670          0     702345          0      24257
 12    4535451       1002          0          0          0          0        501        501          0          0          0       5991
 13   13425648       3478          0          0         54         54       1685       1685          0          0          0      22832
 14    5214701       1310          0          0          1          1        654        654          0          0          0      16212
 15   32683487       8122          0          0         94         94       3967       3967          0          0          0      10940
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.707s (  749.291 blocks/s,  46.831 Mb/s)
  654   Tue Jun 11 23:13:44 2024 TD00:00-08:00 Wednesday 11 June
00.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 3

      ADC data items stats OK - attachment 2
   
      per FEE64 Rate spectra - attachment 7

      per p+n FEE64 1.8.L spectra - attachments 6
       aida09 pulser peak width 63 ch FWHM 

      Merger, TapeServer etc - attachments 4-5

      analysis data file S181/R6_274 - attachment 8
       max FEE64 deadtime 19% aida08
       HEC data rate 3.8kHz, LEC data rate 2.258M

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

03.12 Merger, TapeServer etc - attachments 11-12
       before & after Merger restart

      per FEE64 Rate spectra - attachment 13

      ADC data items stats OK - attachment 14
       aida02 zero rate

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage current OK - attachment 16

      analysis data file S181/R6_372 - attachment 17
       max FEE64 deadtime 22% aida08
       HEC data rate 4.4kHz, LEC data rate 2.295M

03.32 all histograms zero'd

03.57 per p+n FEE64 1.8.L spectra - attachment 18
       25 mins = 1500 @ 2Hz = 3000 cf. peak area c. 3000 => deadtime low
Attachment 1: Screenshot_from_2024-06-12_00-08-05.png
Screenshot_from_2024-06-12_00-08-05.png
Attachment 2: Screenshot_from_2024-06-12_00-11-04.png
Screenshot_from_2024-06-12_00-11-04.png
Attachment 3: Screenshot_from_2024-06-12_00-08-40.png
Screenshot_from_2024-06-12_00-08-40.png
Attachment 4: Screenshot_from_2024-06-12_00-13-13.png
Screenshot_from_2024-06-12_00-13-13.png
Attachment 5: Screenshot_from_2024-06-12_00-12-56.png
Screenshot_from_2024-06-12_00-12-56.png
Attachment 6: Screenshot_from_2024-06-12_00-12-08.png
Screenshot_from_2024-06-12_00-12-08.png
Attachment 7: Screenshot_from_2024-06-12_00-15-36.png
Screenshot_from_2024-06-12_00-15-36.png
Attachment 8: R6_274
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258999041 ( 2257638.6 Hz)
        Other data format:    2920959 (   25461.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1511 (      13.2 Hz)
                               RESUME:       1510 (      13.2 Hz)
                              SYNC100:      32427 (     282.7 Hz)
                              WR48-63:      32427 (     282.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2853084 (   24869.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     439283 (    3829.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      114.721 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.601                0.000
  1                7.539                0.000
  2                0.326                0.000
  3               16.402                0.000
  4                0.635                0.000
  5               17.425                0.000
  6                0.097                0.000
  7               21.859                0.000
  8                0.128                0.000
  9                3.673                0.000
 10                3.675                0.000
 11                0.154                0.000
 12                0.891                0.000
 13                0.524                0.000
 14                0.022                0.000
 15                2.309                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15369403       7472          0          0         46         46       1974       1974          0       3432          0      22292
  1   15769502       4294          0          0        112        112       2035       2035          0          0          0      29638
  2   14433962    1024458          0          0         14         14       1879       1879          0    1020672          0      10278
  3   20761922       5804          0          0        295        295       2607       2607          0          0          0      20061
  4    9340460     275709          0          0         19         19       1175       1175          0     273321          0       7943
  5   21823754       5845          0          0        181        180       2742       2742          0          0          0      18704
  6    5422542     230099          0          0          5          5        701        701          0     228687          0       9316
  7   37349907       9452          0          0        244        244       4482       4482          0          0          0     214200
  8    5081034       1202          0          0          1          1        600        600          0          0          0      10181
  9   31593425     999213          0          0        129        129       4047       4047          0     990861          0       8738
 10   17180710       4686          0          0        246        246       2097       2097          0          0          0       5196
 11    7163253     337989          0          0         11         11        928        928          0     336111          0      29914
 12    8343534       2202          0          0         50         50       1051       1051          0          0          0       7185
 13   14807749       3744          0          0         49         49       1823       1823          0          0          0      15937
 14    5467474       1324          0          0          3          3        659        659          0          0          0       5944
 15   29090410       7466          0          0        106        106       3627       3627          0          0          0      23756
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.896s (  867.303 blocks/s,  54.206 Mb/s)
Attachment 9: Screenshot_from_2024-06-12_00-24-11.png
Screenshot_from_2024-06-12_00-24-11.png
Attachment 10: Screenshot_from_2024-06-12_00-23-16.png
Screenshot_from_2024-06-12_00-23-16.png
Attachment 11: Screenshot_from_2024-06-12_03-11-52.png
Screenshot_from_2024-06-12_03-11-52.png
Attachment 12: Screenshot_from_2024-06-12_03-05-47.png
Screenshot_from_2024-06-12_03-05-47.png
Attachment 13: Screenshot_from_2024-06-12_03-04-36.png
Screenshot_from_2024-06-12_03-04-36.png
Attachment 14: Screenshot_from_2024-06-12_03-04-13.png
Screenshot_from_2024-06-12_03-04-13.png
Attachment 15: Screenshot_from_2024-06-12_03-02-04.png
Screenshot_from_2024-06-12_03-02-04.png
Attachment 16: Screenshot_from_2024-06-12_03-01-26.png
Screenshot_from_2024-06-12_03-01-26.png
Attachment 17: R6_372
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260609951 ( 2295050.9 Hz)
        Other data format:    1310049 (   11536.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1765 (      15.5 Hz)
                               RESUME:       1765 (      15.5 Hz)
                              SYNC100:      32423 (     285.5 Hz)
                              WR48-63:      32423 (     285.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1241673 (   10934.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     498232 (    4387.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      113.553 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.716                0.000
  1                0.000                0.000
  2                0.675                0.000
  3               20.755                0.000
  4                8.029                0.000
  5               21.275                0.000
  6                0.025                0.000
  7               25.466                0.000
  8                0.004                0.000
  9                3.222                0.000
 10                3.079                0.000
 11                0.541                0.000
 12                3.417                0.000
 13                0.637                0.000
 14                0.101                0.000
 15                2.523                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16545027       7593          0          0         61         61       2039       2039          0       3393          0      34807
  1          0          0          0          0          0          0          0          0          0          0          0          0
  2   16018011       4126          0          0         31         31       2032       2032          0          0          0      16400
  3   22563914       6244          0          0        367        367       2755       2755          0          0          0      27855
  4   14584632     380495          0          0        101        101       1904       1904          0     376485          0      10790
  5   23574124       6600          0          0        239        239       3061       3061          0          0          0      28773
  6    6172794     333349          0          0          6          6        797        797          0     331743          0      14869
  7   36993806       9704          0          0        287        287       4565       4565          0          0          0     198994
  8    6095601       1620          0          0          1          1        809        809          0          0          0      16000
  9   30899077       7806          0          0        122        122       3781       3781          0          0          0      13820
 10   16520099       4484          0          0        262        262       1980       1980          0          0          0       8141
 11   10197807     532664          0          0         39         39       1267       1267          0     530052          0      46994
 12   11352337       3042          0          0        119        119       1402       1402          0          0          0      10315
 13   14645959       3712          0          0         38         38       1818       1818          0          0          0      24695
 14    5727639       1460          0          0          7          7        723        723          0          0          0       9258
 15   28719124       7150          0          0         85         85       3490       3490          0          0          0      36521
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   37.054s (  863.611 blocks/s,  53.976 Mb/s)
Attachment 18: Screenshot_from_2024-06-12_03-57-50.png
Screenshot_from_2024-06-12_03-57-50.png
  653   Tue Jun 11 20:20:36 2024 TDADC data rates from 09.00 11.6.24
Per NH

"From ELOG (assuming times accurateish)

10:00 Source in (AIDA removed from DESPEC DAQ)
11:30: Source out (?)
11:45: Background Runs
13:00: bPlast thresholds changed
15:45: Beam Returned"
-
                  ADC data item rate
EOF time  file    LEC        HEC
09.02    R4_538   0.906M     1.7k
10.02      _551   0.927M     1.7k
11.00      _564   0.875M     1.7k
11.19      _568   0.872M     1.7k
12.18    R6_7     1.121M     1.9k
13.03     6_19    1.134M     1.9k
14.02     6_35    1.197M     2.0k
15.00     6_52    1.232M     2.0k
15.10     6_54    1.225M     2.0k
15.36     6_56    1.467M     2.0k
16.00     6_65    1.630M     3.0k
17.01     6_89    1.778M     3.7k
21.43     6_207   1.759M     2.7k
12.6.24
00.09     6_274   2.258M     3.8k
03.12     6_372   2.295M     4.4k
07.13     6_506   2.284M     4.5k
11.12     6_611   1.853M     3.6k
14.55     7_16    1.877M     4.7k
13.6.24
07.01     7_457   1.788M     4.3k
12.34     7_610   2.000M     4.7k
14.00     7_649   1.756M     2.5k
14.12     7_654   1.817M     2.5k
15.33     7_693   1.907M     3.5k
14.6.24
07.23     8_35    1.677M     3.7k
10.54     8_122   1.880M     4.7k
14.38     8_213   1.935M     3.6k
15.6.24
01.25     9_27    0.951k     1.6k  (beam off from c. 18.15 14.6.24)
08.56     9_152   1.215M     2.1k  gamma source in use?
15.27     9_275   1.520M     2.3k  370kBq 152Eu source in use
16.6.24
15.06     9_747   1.667M     2.3k

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 
  652   Tue Jun 11 18:45:37 2024 PPMid-shift checks, 19:45

All looks good.

Sceenshots attached.

Attachment 1: Screenshot_from_2024-06-11_19-46-41.png
Screenshot_from_2024-06-11_19-46-41.png
Attachment 2: Screenshot_from_2024-06-11_19-47-09.png
Screenshot_from_2024-06-11_19-47-09.png
Attachment 3: Screenshot_from_2024-06-11_19-47-38.png
Screenshot_from_2024-06-11_19-47-38.png
Attachment 4: Screenshot_from_2024-06-11_19-48-12.png
Screenshot_from_2024-06-11_19-48-12.png
Attachment 5: Screenshot_from_2024-06-11_19-48-58.png
Screenshot_from_2024-06-11_19-48-58.png
Attachment 6: Screenshot_from_2024-06-11_19-49-22.png
Screenshot_from_2024-06-11_19-49-22.png
Attachment 7: Screenshot_from_2024-06-11_19-49-43.png
Screenshot_from_2024-06-11_19-49-43.png
Attachment 8: Screenshot_from_2024-06-11_19-50-19.png
Screenshot_from_2024-06-11_19-50-19.png
  651   Tue Jun 11 15:01:07 2024 PP TD16:00-00:00 shift Tuesday 11/06/24
All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649
Attachment 1: Screenshot_from_2024-06-11_16-06-43.png
Screenshot_from_2024-06-11_16-06-43.png
Attachment 2: Screenshot_from_2024-06-11_16-07-53.png
Screenshot_from_2024-06-11_16-07-53.png
Attachment 3: Screenshot_from_2024-06-11_16-09-03.png
Screenshot_from_2024-06-11_16-09-03.png
Attachment 4: Screenshot_from_2024-06-11_16-12-05.png
Screenshot_from_2024-06-11_16-12-05.png
Attachment 5: Screenshot_from_2024-06-11_16-12-46.png
Screenshot_from_2024-06-11_16-12-46.png
Attachment 6: Screenshot_from_2024-06-11_16-13-40.png
Screenshot_from_2024-06-11_16-13-40.png
Attachment 7: Screenshot_from_2024-06-11_16-14-11.png
Screenshot_from_2024-06-11_16-14-11.png
Attachment 8: Screenshot_from_2024-06-11_16-17-21.png
Screenshot_from_2024-06-11_16-17-21.png
Attachment 9: Screenshot_from_2024-06-11_17-34-26.png
Screenshot_from_2024-06-11_17-34-26.png
Attachment 10: Screenshot_from_2024-06-11_17-33-33.png
Screenshot_from_2024-06-11_17-33-33.png
Attachment 11: Screenshot_from_2024-06-11_17-39-32.png
Screenshot_from_2024-06-11_17-39-32.png
Attachment 12: Screenshot_from_2024-06-11_17-48-43.png
Screenshot_from_2024-06-11_17-48-43.png
Attachment 13: R6_98
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258260025 ( 1785474.1 Hz)
        Other data format:    3659975 (   25303.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        939 (       6.5 Hz)
                               RESUME:        938 (       6.5 Hz)
                              SYNC100:      32538 (     225.0 Hz)
                              WR48-63:      32538 (     225.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3593022 (   24840.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     539819 (    3732.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.645 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.468                0.000
  1                0.015                0.000
  2                0.040                0.000
  3                0.872                0.000
  4                0.007                0.000
  5                0.312                0.000
  6                0.000                0.000
  7                7.299                0.000
  8                0.001                0.000
  9                3.731                0.000
 10                2.112                0.000
 11                0.008                0.000
 12                0.000                0.000
 13                0.367                0.000
 14                0.001                0.000
 15                1.636                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15794317       8531          0          0         49         49       2058       2058          0       4317          0      10696
  1    7121332       1752          0          0          3          3        873        873          0          0          0      24880
  2   15560192    1259782          0          0          6          6       2078       2078          0    1255614          0      14645
  3   17271150       4664          0          0        134        134       2198       2198          0          0          0      19004
  4    6664272     367237          0          0          2          2        840        840          0     365553          0       5226
  5   16529211       4220          0          0         48         48       2062       2062          0          0          0      23130
  6    5459932     300460          0          0          0          0        680        680          0     299100          0      14450
  7   45345978      11675          0          0        200        199       5638       5638          0          0          0     336191
  8    6229388       1534          0          0          1          1        766        766          0          0          0      16825
  9   38606423    1232596          0          0        155        155       5043       5043          0    1222200          0      15969
 10   19820101       5140          0          0        195        195       2375       2375          0          0          0       9560
 11    4555885     447544          0          0          2          2        651        651          0     446238          0      14419
 12    4015114        976          0          0          0          0        488        488          0          0          0       3706
 13   15417116       3872          0          0         46         46       1890       1890          0          0          0      14088
 14    5845488       1420          0          0          1          1        709        709          0          0          0      10117
 15   34024126       8572          0          0         97         97       4189       4189          0          0          0       6913
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.520s (  876.243 blocks/s,  54.765 Mb/s)
  650   Tue Jun 11 10:03:48 2024 TD, NH08:00-16:00 Tuesday 10 June
10.57 DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2

      ADC, MBS correlation scaler data item stats - attachments 3-4
       ADC 9x < 20k, max 243k aida08

      ASIC settings aida01 & aida02 - attachments 5-6

      ALL system wide checks OK *except* attachments 7-8

      WR timestamps OK - attachment 9

      BNC PB-5 settings - attachment 10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L spectra - attachment 12
       aida09 pulser peak width 57 ch FWHM

      per FEE64 1.8.H spectra - attachments 13-14

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16
       no p+n FEE64 wavefroms - reason unknown

      Merger, TapeServer etc - attachments 17-18

11.25 Lost control of aida-gsi window system 
      All command terminals lost, could not start new command terminals

      aida-gsi restart

      FEE64 power cycle
      DSSSD bias cycle

      Note file manager, calculator apps now working as expected

12.11 DAQ starts 
      data file S181/R6

      ASIC settings 2024Apr19-15.22.49

      BNC PB-5 Pulser
      Amplitude 10.0V
      Attenuation x10
      Fall time 1ms
      Frequency 2Hz
      Polarity + (connected to test + signal daisy chain to all p+n FEE64s)

      Sync ASIC clocks OK
      All system wide checks OK *except* aida02 FPGA/WR decoder - attachments 19-20

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-22

12.29 analysis data file S181/R6_9 - attachment 23
      max dead time c. 2% (aida08)

15:34 For some reason the AIDA MBS wasn't connecting to the time sorter (or a check stream) [NH]
      Try to reset the AIDA MBS 
      Due to confusion the AIDA Merger was rebooted too, while the DAQ was running, this broke all the FEEs -> Powecycled
      Synced ASIC clocks, fixed all waveforms
      Masked all discriminators
      All looks ok -> Sending back to MBS and back in DESPEC DAQ
Attachment 1: Screenshot_from_2024-06-11_10-56-57.png
Screenshot_from_2024-06-11_10-56-57.png
Attachment 2: Screenshot_from_2024-06-11_10-57-13.png
Screenshot_from_2024-06-11_10-57-13.png
Attachment 3: Screenshot_from_2024-06-11_10-57-29.png
Screenshot_from_2024-06-11_10-57-29.png
Attachment 4: Screenshot_from_2024-06-11_10-57-44.png
Screenshot_from_2024-06-11_10-57-44.png
Attachment 5: Screenshot_from_2024-06-11_11-02-06.png
Screenshot_from_2024-06-11_11-02-06.png
Attachment 6: Screenshot_from_2024-06-11_11-02-12.png
Screenshot_from_2024-06-11_11-02-12.png
Attachment 7: Screenshot_from_2024-06-11_11-02-29.png
Screenshot_from_2024-06-11_11-02-29.png
Attachment 8: Screenshot_from_2024-06-11_11-02-33.png
Screenshot_from_2024-06-11_11-02-33.png
Attachment 9: Screenshot_from_2024-06-11_11-02-57.png
Screenshot_from_2024-06-11_11-02-57.png
Attachment 10: Screenshot_from_2024-06-11_11-03-07.png
Screenshot_from_2024-06-11_11-03-07.png
Attachment 11: Screenshot_from_2024-06-11_10-57-57.png
Screenshot_from_2024-06-11_10-57-57.png
Attachment 12: Screenshot_from_2024-06-11_10-59-40.png
Screenshot_from_2024-06-11_10-59-40.png
Attachment 13: Screenshot_from_2024-06-11_11-00-40.png
Screenshot_from_2024-06-11_11-00-40.png
Attachment 14: Screenshot_from_2024-06-11_11-01-11.png
Screenshot_from_2024-06-11_11-01-11.png
Attachment 15: Screenshot_from_2024-06-11_11-01-27.png
Screenshot_from_2024-06-11_11-01-27.png
Attachment 16: Screenshot_from_2024-06-11_11-01-52.png
Screenshot_from_2024-06-11_11-01-52.png
Attachment 17: Screenshot_from_2024-06-11_11-03-24.png
Screenshot_from_2024-06-11_11-03-24.png
Attachment 18: Screenshot_from_2024-06-11_11-03-33.png
Screenshot_from_2024-06-11_11-03-33.png
Attachment 19: Screenshot_from_2024-06-11_12-18-20.png
Screenshot_from_2024-06-11_12-18-20.png
Attachment 20: Screenshot_from_2024-06-11_12-18-15.png
Screenshot_from_2024-06-11_12-18-15.png
Attachment 21: Screenshot_from_2024-06-11_12-23-35.png
Screenshot_from_2024-06-11_12-23-35.png
Attachment 22: Screenshot_from_2024-06-11_12-22-13.png
Screenshot_from_2024-06-11_12-22-13.png
Attachment 23: R6_9
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258839908 ( 1138557.8 Hz)
        Other data format:    3080092 (   13548.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        641 (       2.8 Hz)
                               RESUME:        641 (       2.8 Hz)
                              SYNC100:      32847 (     144.5 Hz)
                              WR48-63:      32847 (     144.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3013116 (   13253.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     428993 (    1887.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      227.340 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.006                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.570                0.000
  4                0.000                0.000
  5                0.059                0.000
  6                0.000                0.000
  7                4.510                0.000
  8                0.000                0.000
  9                2.812                0.000
 10                0.098                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.022                0.000
 14                0.000                0.000
 15                1.041                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8222140       8845          0          0          2          2       1011       1011          0       6819          0          0
  1    3606751        920          0          0          0          0        460        460          0          0          0          0
  2   10679748    1508816          0          0          0          0       1558       1558          0    1505700          0          0
  3   19436638       5222          0          0        115        115       2496       2496          0          0          0          0
  4    3900385        924          0          0          0          0        462        462          0          0          0          0
  5   15462846       3992          0          0         26         26       1970       1970          0          0          0          0
  6    5052319      14939          0          0          0          0        649        649          0      13641          0          0
  7   59921630      15444          0          0        211        211       7511       7511          0          0          0     428993
  8    6654004       1590          0          0          0          0        795        795          0          0          0          0
  9   48245181    1499946          0          0        182        182       6313       6313          0    1486956          0          0
 10   16829911       4196          0          0         17         17       2081       2081          0          0          0          0
 11    2314011        558          0          0          0          0        279        279          0          0          0          0
 12    3820239        906          0          0          0          0        453        453          0          0          0          0
 13   11619760       2934          0          0          7          7       1460       1460          0          0          0          0
 14    3764196        930          0          0          0          0        465        465          0          0          0          0
 15   39310149       9930          0          0         81         81       4884       4884          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.500s ( 1084.746 blocks/s,  67.797 Mb/s)
  649   Mon Jun 10 23:36:14 2024 TD0000-08:00 Tuesday 11 June
00.09 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      per p+n FEE64 1.8.L spectra - attachments 5-6
       aida09 pulser peak width 53 ch FWHM 

      per FEE64 1.8.H spectra - attachments 7-8

      per n+n FEE64 1.8.W spectra - 20us FSR - attachment 9
       p+n 1.8.W not running currently

      Merger, TapeServer etc - attachments 10-11

      analysis data file S181/R4_427 - attachment 12
       max FEE64 deadtime 1.7%
       HEC data rate 3.1kHz

00.28 all histograms zero'd


02.39 DAQ continues data file S181/R4_460
      Beam off since 02.08

      DSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data items stats OK - attachment 15
       9x < 20k, max 239k aida08
   
      per FEE64 Rate spectra - attachment 16


07.15 DAQ continues data file S181/R4_517
      Beam off since 02.08
      HKR report c 3-4 h until UNILAC fixed

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18

      ADC data items stats OK - attachment 19
       8x < 20k, max 238k aida08
   
      per FEE64 Rate spectra - attachment 20
Attachment 1: Screenshot_from_2024-06-11_00-09-43.png
Screenshot_from_2024-06-11_00-09-43.png
Attachment 2: Screenshot_from_2024-06-11_00-10-36.png
Screenshot_from_2024-06-11_00-10-36.png
Attachment 3: Screenshot_from_2024-06-11_00-11-13.png
Screenshot_from_2024-06-11_00-11-13.png
Attachment 4: Screenshot_from_2024-06-11_00-11-45.png
Screenshot_from_2024-06-11_00-11-45.png
Attachment 5: Screenshot_from_2024-06-11_00-13-25.png
Screenshot_from_2024-06-11_00-13-25.png
Attachment 6: Screenshot_from_2024-06-11_00-16-14.png
Screenshot_from_2024-06-11_00-16-14.png
Attachment 7: Screenshot_from_2024-06-11_00-24-51.png
Screenshot_from_2024-06-11_00-24-51.png
Attachment 8: Screenshot_from_2024-06-11_00-25-44.png
Screenshot_from_2024-06-11_00-25-44.png
Attachment 9: Screenshot_from_2024-06-11_00-27-29.png
Screenshot_from_2024-06-11_00-27-29.png
Attachment 10: Screenshot_from_2024-06-11_00-29-21.png
Screenshot_from_2024-06-11_00-29-21.png
Attachment 11: Screenshot_from_2024-06-11_00-29-41.png
Screenshot_from_2024-06-11_00-29-41.png
Attachment 12: R4_427
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256031206 ( 1090943.7 Hz)
        Other data format:    5888794 (   25092.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        689 (       2.9 Hz)
                               RESUME:        689 (       2.9 Hz)
                              SYNC100:      32874 (     140.1 Hz)
                              WR48-63:      32874 (     140.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5821668 (   24806.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     725584 (    3091.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      234.688 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.082                0.000
  1                0.081                0.000
  2                0.099                0.000
  3                1.580                0.000
  4                0.860                0.000
  5                0.325                0.000
  6                0.000                0.000
  7                4.102                0.000
  8                0.020                0.000
  9                2.330                0.000
 10                0.170                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.981                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3851535       8075          0          0          3          3        517        517          0       7035          0      14311
  1   10547580       2760          0          0          9          9       1371       1371          0          0          0      35298
  2    7097044    2015426          0          0          1          1       1095       1095          0    2013234          0      19431
  3   24895003       6544          0          0        192        192       3080       3080          0          0          0      27887
  4    3219755     585586          0          0          4          4        493        493          0     584592          0      17915
  5   19996310       5104          0          0         46         46       2506       2506          0          0          0      32222
  6    5825374     446331          0          0          0          0        741        741          0     444849          0      18892
  7   61314555      15906          0          0        191        191       7762       7762          0          0          0     429462
  8    8374860       2050          0          0          1          1       1024       1024          0          0          0      21845
  9   43291179    2005473          0          0        168        168       5805       5805          0    1993527          0      21309
 10   13902520       3510          0          0          6          6       1749       1749          0          0          0      12563
 11    2813051     779317          0          0          0          0        443        443          0     778431          0      21093
 12    5093380       1266          0          0          0          0        633        633          0          0          0      11468
 13    7142579       1774          0          0          0          0        887        887          0          0          0      18724
 14    3712134        902          0          0          0          0        451        451          0          0          0      12892
 15   34954347       8770          0          0         68         68       4317       4317          0          0          0      10272
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.627s (  924.132 blocks/s,  57.758 Mb/s)
Attachment 13: Screenshot_from_2024-06-11_02-36-59.png
Screenshot_from_2024-06-11_02-36-59.png
Attachment 14: Screenshot_from_2024-06-11_02-41-01.png
Screenshot_from_2024-06-11_02-41-01.png
Attachment 15: Screenshot_from_2024-06-11_02-41-50.png
Screenshot_from_2024-06-11_02-41-50.png
Attachment 16: Screenshot_from_2024-06-11_02-42-40.png
Screenshot_from_2024-06-11_02-42-40.png
Attachment 17: Screenshot_from_2024-06-11_07-13-28.png
Screenshot_from_2024-06-11_07-13-28.png
Attachment 18: Screenshot_from_2024-06-11_07-15-41.png
Screenshot_from_2024-06-11_07-15-41.png
Attachment 19: Screenshot_from_2024-06-11_07-16-08.png
Screenshot_from_2024-06-11_07-16-08.png
Attachment 20: Screenshot_from_2024-06-11_07-16-35.png
Screenshot_from_2024-06-11_07-16-35.png
  648   Mon Jun 10 15:07:24 2024 Marc16:00-00:00 shift Monday 10/06/24

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7
7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

 

Attachment 1: Screenshot_from_2024-06-10_16-54-55.png
Screenshot_from_2024-06-10_16-54-55.png
Attachment 2: Screenshot_from_2024-06-10_16-55-40.png
Screenshot_from_2024-06-10_16-55-40.png
Attachment 3: Screenshot_from_2024-06-10_17-01-34.png
Screenshot_from_2024-06-10_17-01-34.png
Attachment 4: Screenshot_from_2024-06-10_17-02-22.png
Screenshot_from_2024-06-10_17-02-22.png
Attachment 5: Screenshot_from_2024-06-10_17-05-47.png
Screenshot_from_2024-06-10_17-05-47.png
Attachment 6: Screenshot_from_2024-06-10_17-06-52.png
Screenshot_from_2024-06-10_17-06-52.png
Attachment 7: Screenshot_from_2024-06-10_17-13-25.png
Screenshot_from_2024-06-10_17-13-25.png
Attachment 8: Screenshot_from_2024-06-10_18-59-09.png
Screenshot_from_2024-06-10_18-59-09.png
Attachment 9: Screenshot_from_2024-06-10_18-59-42.png
Screenshot_from_2024-06-10_18-59-42.png
Attachment 10: Screenshot_from_2024-06-10_19-02-24.png
Screenshot_from_2024-06-10_19-02-24.png
Attachment 11: Screenshot_from_2024-06-10_19-01-43.png
Screenshot_from_2024-06-10_19-01-43.png
Attachment 12: Screenshot_from_2024-06-10_19-03-42.png
Screenshot_from_2024-06-10_19-03-42.png
Attachment 13: Screenshot_from_2024-06-10_19-04-24.png
Screenshot_from_2024-06-10_19-04-24.png
Attachment 14: Screenshot_from_2024-06-10_19-05-59.png
Screenshot_from_2024-06-10_19-05-59.png
Attachment 15: Screenshot_from_2024-06-10_19-07-37.png
Screenshot_from_2024-06-10_19-07-37.png
Attachment 16: Screenshot_from_2024-06-10_20-56-43.png
Screenshot_from_2024-06-10_20-56-43.png
Attachment 17: Screenshot_from_2024-06-10_20-57-19.png
Screenshot_from_2024-06-10_20-57-19.png
Attachment 18: Screenshot_from_2024-06-10_20-59-07.png
Screenshot_from_2024-06-10_20-59-07.png
Attachment 19: Screenshot_from_2024-06-10_20-59-34.png
Screenshot_from_2024-06-10_20-59-34.png
Attachment 20: Screenshot_from_2024-06-10_21-00-23.png
Screenshot_from_2024-06-10_21-00-23.png
Attachment 21: Screenshot_from_2024-06-10_21-02-53.png
Screenshot_from_2024-06-10_21-02-53.png
Attachment 22: Screenshot_from_2024-06-10_21-02-04.png
Screenshot_from_2024-06-10_21-02-04.png
Attachment 23: Screenshot_from_2024-06-10_21-08-56.png
Screenshot_from_2024-06-10_21-08-56.png
Attachment 24: Screenshot_from_2024-06-10_22-57-04.png
Screenshot_from_2024-06-10_22-57-04.png
Attachment 25: Screenshot_from_2024-06-10_22-57-32.png
Screenshot_from_2024-06-10_22-57-32.png
Attachment 26: Screenshot_from_2024-06-10_22-58-10.png
Screenshot_from_2024-06-10_22-58-10.png
Attachment 27: Screenshot_from_2024-06-10_22-58-48.png
Screenshot_from_2024-06-10_22-58-48.png
Attachment 28: Screenshot_from_2024-06-10_22-59-34.png
Screenshot_from_2024-06-10_22-59-34.png
Attachment 29: Screenshot_from_2024-06-10_23-00-43.png
Screenshot_from_2024-06-10_23-00-43.png
Attachment 30: Screenshot_from_2024-06-10_23-01-18.png
Screenshot_from_2024-06-10_23-01-18.png
Attachment 31: Screenshot_from_2024-06-10_23-03-00.png
Screenshot_from_2024-06-10_23-03-00.png
  647   Mon Jun 10 07:15:48 2024 Dan08.00-16.00 Monday 10th June

8am checks

temps ok - attachment 2

rates ok - attachment 8

voltage ok - attachment 5

merger ok - attachment  6

 

9am checks

temps ok - attachment 10

rates ok - attachment 9

voltage ok - attachment 12

merger ok - attachment  11

 

(Ignore  Grafana plots 13-43 inc - not refreshed)

 

9.16 beam stopped to optimise beam intensity

 

10.00 checks - still no beam

temps ok - attachment 17

rates ok - attachment 15+16

voltage ok - attachment 14

merger ok - attachment  18

 

11.00 checks - still no beam

temps ok - attachment 23

rates ok - attachment 21+22

voltage ok - attachment 20

merger ok - attachment  24

 

12pm checks

temps ok - attachment 28

rates ok - attachment 26+27

voltage ok - attachment 29

merger ok - attachment  30

 

12.45 - beam back

 

temps ok - attachment 34

rates ok - attachment 32+33

voltage ok - attachment 35

merger ok - attachment  36

 

2pm checks

temps ok - attachment 40

rates ok - attachment 38+39

voltage ok - attachment 41

merger ok - attachment  42

 

/tmp/R4_276 added - attachment 44

max deadtime 10.5% (aida08)

n+n FEE64s deadtime 

 

3pm checks

temps ok - attachment 48

rates ok - attachment 46+47

voltage ok - attachment 45+50

merger ok - attachment  49

Attachment 1: Screenshot_from_2024-06-10_08-07-10.png
Screenshot_from_2024-06-10_08-07-10.png
Attachment 2: Screenshot_from_2024-06-10_08-07-54.png
Screenshot_from_2024-06-10_08-07-54.png
Attachment 3: Screenshot_from_2024-06-10_08-11-27.png
Screenshot_from_2024-06-10_08-11-27.png
Attachment 4: Screenshot_from_2024-06-10_08-13-25.png
Screenshot_from_2024-06-10_08-13-25.png
Attachment 5: Screenshot_from_2024-06-10_08-22-19.png
Screenshot_from_2024-06-10_08-22-19.png
Attachment 6: Screenshot_from_2024-06-10_08-28-14.png
Screenshot_from_2024-06-10_08-28-14.png
Attachment 7: Screenshot_from_2024-06-10_08-33-04.png
Screenshot_from_2024-06-10_08-33-04.png
Attachment 8: Screenshot_from_2024-06-10_09-00-04.png
Screenshot_from_2024-06-10_09-00-04.png
Attachment 9: Screenshot_from_2024-06-10_09-00-35.png
Screenshot_from_2024-06-10_09-00-35.png
Attachment 10: Screenshot_from_2024-06-10_09-01-05.png
Screenshot_from_2024-06-10_09-01-05.png
Attachment 11: Screenshot_from_2024-06-10_09-03-04.png
Screenshot_from_2024-06-10_09-03-04.png
Attachment 12: Screenshot_from_2024-06-10_09-03-40.png
Screenshot_from_2024-06-10_09-03-40.png
Attachment 13: Screenshot_from_2024-06-10_09-04-27.png
Screenshot_from_2024-06-10_09-04-27.png
Attachment 14: Screenshot_from_2024-06-10_10-04-59.png
Screenshot_from_2024-06-10_10-04-59.png
Attachment 15: Screenshot_from_2024-06-10_10-05-36.png
Screenshot_from_2024-06-10_10-05-36.png
Attachment 16: Screenshot_from_2024-06-10_10-06-07.png
Screenshot_from_2024-06-10_10-06-07.png
Attachment 17: Screenshot_from_2024-06-10_10-06-40.png
Screenshot_from_2024-06-10_10-06-40.png
Attachment 18: Screenshot_from_2024-06-10_10-07-24.png
Screenshot_from_2024-06-10_10-07-24.png
Attachment 19: Screenshot_from_2024-06-10_10-09-01.png
Screenshot_from_2024-06-10_10-09-01.png
Attachment 20: Screenshot_from_2024-06-10_11-00-43.png
Screenshot_from_2024-06-10_11-00-43.png
Attachment 21: Screenshot_from_2024-06-10_11-01-36.png
Screenshot_from_2024-06-10_11-01-36.png
Attachment 22: Screenshot_from_2024-06-10_11-02-19.png
Screenshot_from_2024-06-10_11-02-19.png
Attachment 23: Screenshot_from_2024-06-10_11-03-06.png
Screenshot_from_2024-06-10_11-03-06.png
Attachment 24: Screenshot_from_2024-06-10_11-03-58.png
Screenshot_from_2024-06-10_11-03-58.png
Attachment 25: Screenshot_from_2024-06-10_11-04-27.png
Screenshot_from_2024-06-10_11-04-27.png
Attachment 26: Screenshot_from_2024-06-10_12-03-07.png
Screenshot_from_2024-06-10_12-03-07.png
Attachment 27: Screenshot_from_2024-06-10_12-03-36.png
Screenshot_from_2024-06-10_12-03-36.png
Attachment 28: Screenshot_from_2024-06-10_12-04-16.png
Screenshot_from_2024-06-10_12-04-16.png
Attachment 29: Screenshot_from_2024-06-10_12-04-48.png
Screenshot_from_2024-06-10_12-04-48.png
Attachment 30: Screenshot_from_2024-06-10_12-05-27.png
Screenshot_from_2024-06-10_12-05-27.png
Attachment 31: Screenshot_from_2024-06-10_12-06-06.png
Screenshot_from_2024-06-10_12-06-06.png
Attachment 32: Screenshot_from_2024-06-10_12-45-12.png
Screenshot_from_2024-06-10_12-45-12.png
Attachment 33: Screenshot_from_2024-06-10_12-45-39.png
Screenshot_from_2024-06-10_12-45-39.png
Attachment 34: Screenshot_from_2024-06-10_12-46-16.png
Screenshot_from_2024-06-10_12-46-16.png
Attachment 35: Screenshot_from_2024-06-10_12-46-53.png
Screenshot_from_2024-06-10_12-46-53.png
Attachment 36: Screenshot_from_2024-06-10_12-50-29.png
Screenshot_from_2024-06-10_12-50-29.png
Attachment 37: Screenshot_from_2024-06-10_12-51-03.png
Screenshot_from_2024-06-10_12-51-03.png
Attachment 38: Screenshot_from_2024-06-10_13-54-46.png
Screenshot_from_2024-06-10_13-54-46.png
Attachment 39: Screenshot_from_2024-06-10_13-55-20.png
Screenshot_from_2024-06-10_13-55-20.png
Attachment 40: Screenshot_from_2024-06-10_13-55-55.png
Screenshot_from_2024-06-10_13-55-55.png
Attachment 41: Screenshot_from_2024-06-10_13-56-15.png
Screenshot_from_2024-06-10_13-56-15.png
Attachment 42: Screenshot_from_2024-06-10_13-57-11.png
Screenshot_from_2024-06-10_13-57-11.png
Attachment 43: Screenshot_from_2024-06-10_13-57-46.png
Screenshot_from_2024-06-10_13-57-46.png
Attachment 44: R4_276
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257167172 ( 1454701.6 Hz)
        Other data format:    4752828 (   26885.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1210 (       6.8 Hz)
                               RESUME:       1214 (       6.9 Hz)
                              SYNC100:      32658 (     184.7 Hz)
                              WR48-63:      32658 (     184.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4685088 (   26501.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     717535 (    4058.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      176.783 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.207                0.000
  1               13.018                0.000
  2                0.307                0.000
  3               15.029                0.000
  4                0.460                0.000
  5               16.031                0.000
  6                0.053                0.000
  7               18.555                0.000
  8                0.064                0.000
  9                2.767                0.000
 10                0.156                0.000
 11                0.013                0.000
 12                0.042                0.000
 13                0.120                0.000
 14                0.011                0.000
 15                1.165                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6985399       7079          0          0         19         19        870        870          0       5301          0      23500
  1   14902829       3959          0          0        133        134       1846       1846          0          0          0      30106
  2    9849800    1542328          0          0         15         15       1400       1400          0    1539498          0      34705
  3   24099351       6585          0          0        273        274       3019       3019          0          0          0      21224
  4    4358270     608906          0          0         22         22        570        570          0     607722          0      20659
  5   21810185       6107          0          0        208        209       2845       2845          0          0          0      26202
  6    6709059     419440          0          0          8          8        900        900          0     417624          0      34989
  7   50592433      13110          0          0        251        251       6304       6304          0          0          0     314546
  8    8465407       2084          0          0          5          5       1037       1037          0          0          0      35796
  9   38024962    1522758          0          0        170        171       4988       4988          0    1512441          0      34088
 10   15224642       3818          0          0         25         25       1884       1884          0          0          0      21218
 11    4023142     603616          0          0          2          2        555        555          0     602502          0      30177
 12    5915692       1460          0          0          7          7        723        723          0          0          0      19261
 13    9709569       2380          0          0         14         14       1176       1176          0          0          0      33030
 14    4832393       1204          0          0          3          3        599        599          0          0          0      22476
 15   31664039       7994          0          0         55         55       3942       3942          0          0          0      15558
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.457s (  956.451 blocks/s,  59.778 Mb/s)
Attachment 45: Screenshot_from_2024-06-10_15-02-52.png
Screenshot_from_2024-06-10_15-02-52.png
Attachment 46: Screenshot_from_2024-06-10_15-03-47.png
Screenshot_from_2024-06-10_15-03-47.png
Attachment 47: Screenshot_from_2024-06-10_15-04-15.png
Screenshot_from_2024-06-10_15-04-15.png
Attachment 48: Screenshot_from_2024-06-10_15-04-48.png
Screenshot_from_2024-06-10_15-04-48.png
Attachment 49: Screenshot_from_2024-06-10_15-05-18.png
Screenshot_from_2024-06-10_15-05-18.png
Attachment 50: Screenshot_from_2024-06-10_15-15-55.png
Screenshot_from_2024-06-10_15-15-55.png
  646   Sun Jun 9 22:49:56 2024 TD23:00-07:00 Monday 9 June
23.47 DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 4-5 & 9

      WR timestamp - attachment 7

      all system wide checks OK *except* attachments 6 & 10

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-13

      Merger, TapeServer etc - attachments 14-15
       data file S181/R4, TapeServer No Storage mode

00.00 Beam ON 3s spill on, 2s spill off
      99%+ fission fragments
      ADC data item stats - attachment 16


02.03 Data storage enabled
 
      per FEE64 1.8.H spectra - attachments 17-18

02.35 analysis data file S181/R4_28 - attachment 19
      data file S181/R4
      max deadtime aida08 18%
      n+n FEE64s 12-18%, all p+n FEE64s < 2%

02.41 Noted that there is disc data written to disk - all disc data now disabled for all FEE64s

04.08 DSSSD bias & leakage current OK - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Rate spectra - attachment 23

04.15 aida02 no data
      restarted merger per https://elog.ph.ed.ac.uk/DESPEC/644

06.00 DSSSD bias & leakage current OK - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27
Attachment 1: Screenshot_from_2024-06-09_23-47-24.png
Screenshot_from_2024-06-09_23-47-24.png
Attachment 2: Screenshot_from_2024-06-09_23-47-42.png
Screenshot_from_2024-06-09_23-47-42.png
Attachment 3: Screenshot_from_2024-06-09_23-48-00.png
Screenshot_from_2024-06-09_23-48-00.png
Attachment 4: Screenshot_from_2024-06-09_23-48-04.png
Screenshot_from_2024-06-09_23-48-04.png
Attachment 5: Screenshot_from_2024-06-09_23-48-16.png
Screenshot_from_2024-06-09_23-48-16.png
Attachment 6: Screenshot_from_2024-06-09_23-48-41.png
Screenshot_from_2024-06-09_23-48-41.png
Attachment 7: Screenshot_from_2024-06-09_23-49-10.png
Screenshot_from_2024-06-09_23-49-10.png
Attachment 8: Screenshot_from_2024-06-09_23-49-18.png
Screenshot_from_2024-06-09_23-49-18.png
Attachment 9: Screenshot_from_2024-06-09_23-55-16.png
Screenshot_from_2024-06-09_23-55-16.png
Attachment 10: Screenshot_from_2024-06-09_23-55-48.png
Screenshot_from_2024-06-09_23-55-48.png
Attachment 11: Screenshot_from_2024-06-09_23-56-49.png
Screenshot_from_2024-06-09_23-56-49.png
Attachment 12: Screenshot_from_2024-06-09_23-57-05.png
Screenshot_from_2024-06-09_23-57-05.png
Attachment 13: Screenshot_from_2024-06-09_23-57-28.png
Screenshot_from_2024-06-09_23-57-28.png
Attachment 14: Screenshot_from_2024-06-09_23-58-23.png
Screenshot_from_2024-06-09_23-58-23.png
Attachment 15: Screenshot_from_2024-06-09_23-58-29.png
Screenshot_from_2024-06-09_23-58-29.png
Attachment 16: Screenshot_from_2024-06-10_00-11-59.png
Screenshot_from_2024-06-10_00-11-59.png
Attachment 17: Screenshot_from_2024-06-10_01-27-52.png
Screenshot_from_2024-06-10_01-27-52.png
Attachment 18: Screenshot_from_2024-06-10_02-01-38.png
Screenshot_from_2024-06-10_02-01-38.png
Attachment 19: R4_28
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253194077 ( 1696553.0 Hz)
        Other data format:    8725923 (   58468.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1564 (      10.5 Hz)
                               RESUME:       1562 (      10.5 Hz)
                              SYNC100:      32556 (     218.1 Hz)
                              WR48-63:      32556 (     218.1 Hz)
                           FEE64 disc:    3047502 (   20420.1 Hz)
                             MBS info:    5610183 (   37591.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     916603 (    6141.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      149.240 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.269                0.000
  1               18.572                0.000
  2                0.203                0.000
  3               23.804                0.000
  4                0.695                0.000
  5               26.433                0.000
  6                0.080                0.000
  7               27.735                0.000
  8                0.165                0.000
  9                3.128                0.000
 10                0.210                0.000
 11                0.060                0.000
 12                0.056                0.000
 13                0.214                0.000
 14                0.073                0.000
 15                1.249                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8063714     229134          0          0         29         29       1100       1100     222403       4473          0      40546
  1   19451828     341670          0          0        189        189       2480       2480     336332          0          0      50938
  2    9906047    1695573          0          0         32         32       1474       1474     189984    1502577          0      57685
  3   25485973     434336          0          0        375        375       3209       3209     427168          0          0      33926
  4    5225989    1088546          0          0         26         25        750        750      83786    1003209          0      36956
  5   23827850     287577          0          0        303        303       2970       2970     281031          0          0      42461
  6    7632945     803643          0          0         10         10       1067       1067     163557     637932          0      57385
  7   43365063     397279          0          0        289        288       5505       5505     385692          0          0     246600
  8    9556580     143268          0          0         16         16       1175       1175     140886          0          0      58513
  9   31955982    1616616          0          0        155        155       4212       4212     145847    1462035          0      55475
 10   13350940      89761          0          0         23         23       1645       1645      86425          0          0      31430
 11    5509660    1131598          0          0          8          8        744        744     130137     999957          0      52425
 12    7108103     102578          0          0          8          8        887        887     100788          0          0      34624
 13   10337974     150173          0          0         26         26       1313       1313     147495          0          0      56686
 14    5578272     207359          0          0          7          7        687        687     205971          0          0      33046
 15   26837157       6812          0          0         68         68       3338       3338          0          0          0      27907
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.183s (  838.078 blocks/s,  52.380 Mb/s)
Attachment 20: Screenshot_from_2024-06-10_04-08-10.png
Screenshot_from_2024-06-10_04-08-10.png
Attachment 21: Screenshot_from_2024-06-10_04-08-44.png
Screenshot_from_2024-06-10_04-08-44.png
Attachment 22: Screenshot_from_2024-06-10_04-10-54.png
Screenshot_from_2024-06-10_04-10-54.png
Attachment 23: Screenshot_from_2024-06-10_04-11-33.png
Screenshot_from_2024-06-10_04-11-33.png
Attachment 24: Screenshot_from_2024-06-10_06-00-15.png
Screenshot_from_2024-06-10_06-00-15.png
Attachment 25: Screenshot_from_2024-06-10_06-00-52.png
Screenshot_from_2024-06-10_06-00-52.png
Attachment 26: Screenshot_from_2024-06-10_06-01-18.png
Screenshot_from_2024-06-10_06-01-18.png
Attachment 27: Screenshot_from_2024-06-10_06-01-43.png
Screenshot_from_2024-06-10_06-01-43.png
  645   Sun Jun 9 16:38:30 2024 Norah , Muneerah, JB16:0-00:00 9 June 2024

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

 

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

 

 

 

 

Attachment 1: Screenshot_from_2024-06-09_17-24-13.png
Screenshot_from_2024-06-09_17-24-13.png
Attachment 2: Screenshot_from_2024-06-09_18-00-27.png
Screenshot_from_2024-06-09_18-00-27.png
Attachment 3: Screenshot_from_2024-06-09_18-01-49.png
Screenshot_from_2024-06-09_18-01-49.png
Attachment 4: Screenshot_from_2024-06-09_18-01-23.png
Screenshot_from_2024-06-09_18-01-23.png
  644   Sun Jun 9 13:32:48 2024 TD[How To] Recover DAQ when one FEE64 stops sending data - short(er) version
1) Select 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode please ignore this step

2) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

3) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

4) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

5) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

6) Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

8) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

9) Unselect 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode at the start of this procedure please ignore this step


   If you continue to have problems, please request expert assistance.
  643   Sat Jun 8 23:00:52 2024 TDSunday 9 June
23.58 8.6.24

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

03.43 slow comparator 0x64

      DSSSD bias & leakage current OK - attachments 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 1.8.H spectra - attachments 8-9

      per FEE64 Rate spectra - attachment 10


09.11 DSSSD bias & leakage current OK - attachments 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats - attachment 13

      per FEE64 Rate spectra - attachment 14


15.05 online spectra
      per DSSSD HEC E_p versus E_n and x versus y
      observe flat field of c. 500MeV at per spull rate c. 5Hz - fission fragments?
Attachment 1: Screenshot_from_2024-06-08_23-58-55.png
Screenshot_from_2024-06-08_23-58-55.png
Attachment 2: Screenshot_from_2024-06-08_23-59-16.png
Screenshot_from_2024-06-08_23-59-16.png
Attachment 3: Screenshot_from_2024-06-09_00-00-02.png
Screenshot_from_2024-06-09_00-00-02.png
Attachment 4: Screenshot_from_2024-06-09_00-00-32.png
Screenshot_from_2024-06-09_00-00-32.png
Attachment 5: Screenshot_from_2024-06-09_03-43-32.png
Screenshot_from_2024-06-09_03-43-32.png
Attachment 6: Screenshot_from_2024-06-09_03-44-19.png
Screenshot_from_2024-06-09_03-44-19.png
Attachment 7: Screenshot_from_2024-06-09_03-46-03.png
Screenshot_from_2024-06-09_03-46-03.png
Attachment 8: Screenshot_from_2024-06-09_03-46-37.png
Screenshot_from_2024-06-09_03-46-37.png
Attachment 9: Screenshot_from_2024-06-09_03-47-40.png
Screenshot_from_2024-06-09_03-47-40.png
Attachment 10: Screenshot_from_2024-06-09_03-49-06.png
Screenshot_from_2024-06-09_03-49-06.png
Attachment 11: Screenshot_from_2024-06-09_11-15-26.png
Screenshot_from_2024-06-09_11-15-26.png
Attachment 12: Screenshot_from_2024-06-09_11-27-22.png
Screenshot_from_2024-06-09_11-27-22.png
Attachment 13: Screenshot_from_2024-06-09_11-27-52.png
Screenshot_from_2024-06-09_11-27-52.png
Attachment 14: Screenshot_from_2024-06-09_11-28-19.png
Screenshot_from_2024-06-09_11-28-19.png
Attachment 15: Capture_2.PNG
Capture_2.PNG
Attachment 16: Capture.PNG
Capture.PNG
  642   Sat Jun 8 13:22:36 2024 TDSaturday 8 June
14.09 aida02 WR stall
      DAQ STOP
      restart Merger, Mrger SETUP, Merger GO
      DAQ GO

      all histograms & stats zero'd

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 4

      WR timestamp before/after Merger restart - attachments 3 & 5

      all system wide checks OK *except* attachments 6-7

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 8-10
       ADC data items rates high - source installed?

      per FEE64 Rate spectra - attachments 11-13

      per FEE64 1.8.W spectra - 20us FSR - attachments 14-15

      per p+n FEE64 1.8.L spectra - attachment 16
       aida09 pulser peak width 60 ch FWHM

      Merger, TapeServer etc - attachments 17-19
       data file S181/R2_249 switch TapeServer to No Storage mode

      per p+n FEE64 1.8.W spectra - 20us FSR - attachment 20

20.33 DSSSD bias & leakage current OK - attachments 21

      FEE64 temperatures OK - attachment 22

      ADC data item stats - attachment 23

      per FEE64 Rate spectra - attachment 24
Attachment 1: Screenshot_from_2024-06-08_14-07-37.png
Screenshot_from_2024-06-08_14-07-37.png
Attachment 2: Screenshot_from_2024-06-08_14-07-52.png
Screenshot_from_2024-06-08_14-07-52.png
Attachment 3: Screenshot_from_2024-06-08_14-08-47.png
Screenshot_from_2024-06-08_14-08-47.png
Attachment 4: Screenshot_from_2024-06-08_14-08-21.png
Screenshot_from_2024-06-08_14-08-21.png
Attachment 5: Screenshot_from_2024-06-08_14-11-30.png
Screenshot_from_2024-06-08_14-11-30.png
Attachment 6: Screenshot_from_2024-06-08_14-12-20.png
Screenshot_from_2024-06-08_14-12-20.png
Attachment 7: Screenshot_from_2024-06-08_14-12-25.png
Screenshot_from_2024-06-08_14-12-25.png
Attachment 8: Screenshot_from_2024-06-08_14-12-53.png
Screenshot_from_2024-06-08_14-12-53.png
Attachment 9: Screenshot_from_2024-06-08_14-13-12.png
Screenshot_from_2024-06-08_14-13-12.png
Attachment 10: Screenshot_from_2024-06-08_14-13-34.png
Screenshot_from_2024-06-08_14-13-34.png
Attachment 11: Screenshot_from_2024-06-08_14-14-41.png
Screenshot_from_2024-06-08_14-14-41.png
Attachment 12: Screenshot_from_2024-06-08_14-14-48.png
Screenshot_from_2024-06-08_14-14-48.png
Attachment 13: Screenshot_from_2024-06-08_14-14-54.png
Screenshot_from_2024-06-08_14-14-54.png
Attachment 14: Screenshot_from_2024-06-08_14-16-54.png
Screenshot_from_2024-06-08_14-16-54.png
Attachment 15: Screenshot_from_2024-06-08_14-17-49.png
Screenshot_from_2024-06-08_14-17-49.png
Attachment 16: Screenshot_from_2024-06-08_14-20-04.png
Screenshot_from_2024-06-08_14-20-04.png
Attachment 17: Screenshot_from_2024-06-08_14-21-31.png
Screenshot_from_2024-06-08_14-21-31.png
Attachment 18: Screenshot_from_2024-06-08_14-22-05.png
Screenshot_from_2024-06-08_14-22-05.png
Attachment 19: Screenshot_from_2024-06-08_14-22-12.png
Screenshot_from_2024-06-08_14-22-12.png
Attachment 20: Screenshot_from_2024-06-08_14-46-36.png
Screenshot_from_2024-06-08_14-46-36.png
Attachment 21: Screenshot_from_2024-06-08_20-33-56.png
Screenshot_from_2024-06-08_20-33-56.png
Attachment 22: Screenshot_from_2024-06-08_20-36-17.png
Screenshot_from_2024-06-08_20-36-17.png
Attachment 23: Screenshot_from_2024-06-08_20-36-50.png
Screenshot_from_2024-06-08_20-36-50.png
Attachment 24: Screenshot_from_2024-06-08_20-37-34.png
Screenshot_from_2024-06-08_20-37-34.png
  641   Fri Jun 7 16:08:37 2024 TDFriday 7 June
05.01 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      9x < 20k, max 228k aida08

      per FEE64 Rate spectra - attachment 4


08.18 DSSSD bias & leakage current - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 8


14.37 DSSSD bias & leakage current - attachment 9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 12

      Writing to disk file S181/R1 (continuing alpha background file)

14.38 End run S181/R1 and start new run file S181/R2

17.09 08.18 DSSSD bias & leakage current - attachment 13

      FEE64 temperatures OK - attachment 14

      MBS correlation scaler data - attachment 15
       data to aida01, aida02, aida03 and aida10

      ADC data item stats - attachment 16
      9x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 17

      Merger/TapeServer etc - attachments 18-19 
Attachment 1: Screenshot_from_2024-06-07_05-01-53.png
Screenshot_from_2024-06-07_05-01-53.png
Attachment 2: Screenshot_from_2024-06-07_05-00-13.png
Screenshot_from_2024-06-07_05-00-13.png
Attachment 3: Screenshot_from_2024-06-07_05-00-49.png
Screenshot_from_2024-06-07_05-00-49.png
Attachment 4: Screenshot_from_2024-06-07_05-01-23.png
Screenshot_from_2024-06-07_05-01-23.png
Attachment 5: Screenshot_from_2024-06-07_08-18-34.png
Screenshot_from_2024-06-07_08-18-34.png
Attachment 6: Screenshot_from_2024-06-07_08-19-54.png
Screenshot_from_2024-06-07_08-19-54.png
Attachment 7: Screenshot_from_2024-06-07_08-20-21.png
Screenshot_from_2024-06-07_08-20-21.png
Attachment 8: Screenshot_from_2024-06-07_08-19-22.png
Screenshot_from_2024-06-07_08-19-22.png
Attachment 9: Screenshot_from_2024-06-07_14-38-06.png
Screenshot_from_2024-06-07_14-38-06.png
Attachment 10: Screenshot_from_2024-06-07_14-36-36.png
Screenshot_from_2024-06-07_14-36-36.png
Attachment 11: Screenshot_from_2024-06-07_14-37-12.png
Screenshot_from_2024-06-07_14-37-12.png
Attachment 12: Screenshot_from_2024-06-07_14-37-42.png
Screenshot_from_2024-06-07_14-37-42.png
Attachment 13: Screenshot_from_2024-06-07_17-09-16.png
Screenshot_from_2024-06-07_17-09-16.png
Attachment 14: Screenshot_from_2024-06-07_17-09-46.png
Screenshot_from_2024-06-07_17-09-46.png
Attachment 15: Screenshot_from_2024-06-07_17-10-10.png
Screenshot_from_2024-06-07_17-10-10.png
Attachment 16: Screenshot_from_2024-06-07_17-10-52.png
Screenshot_from_2024-06-07_17-10-52.png
Attachment 17: Screenshot_from_2024-06-07_17-11-15.png
Screenshot_from_2024-06-07_17-11-15.png
Attachment 18: Screenshot_from_2024-06-07_17-11-40.png
Screenshot_from_2024-06-07_17-11-40.png
Attachment 19: Screenshot_from_2024-06-07_17-12-21.png
Screenshot_from_2024-06-07_17-12-21.png
  640   Thu Jun 6 11:13:29 2024 TDThursday 6 June
12.09 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      8x < 20k, max 247k aida08

      per FEE64 Rate spectra - attachment 4

12.16 all histograms & stats zero'd


21.08 DSSSD bias & leakage current - attachments 5-6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8
      8x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 9


      
Attachment 1: Screenshot_from_2024-06-06_12-09-27.png
Screenshot_from_2024-06-06_12-09-27.png
Attachment 2: Screenshot_from_2024-06-06_12-11-58.png
Screenshot_from_2024-06-06_12-11-58.png
Attachment 3: Screenshot_from_2024-06-06_12-12-38.png
Screenshot_from_2024-06-06_12-12-38.png
Attachment 4: Screenshot_from_2024-06-06_12-13-09.png
Screenshot_from_2024-06-06_12-13-09.png
Attachment 5: Screenshot_from_2024-06-06_21-08-07.png
Screenshot_from_2024-06-06_21-08-07.png
Attachment 6: Screenshot_from_2024-06-06_21-08-38.png
Screenshot_from_2024-06-06_21-08-38.png
Attachment 7: Screenshot_from_2024-06-06_21-09-15.png
Screenshot_from_2024-06-06_21-09-15.png
Attachment 8: Screenshot_from_2024-06-06_21-10-02.png
Screenshot_from_2024-06-06_21-10-02.png
Attachment 9: Screenshot_from_2024-06-06_21-10-26.png
Screenshot_from_2024-06-06_21-10-26.png
  639   Wed Jun 5 08:34:47 2024 JB. AM, MP, MYDetector bias tests

We are trying to bias each system on its own and in sequence biasing each other detector system to see the interdependence.
 

HV OK. TEMP OK. Attachment 1 &2.

9.43 bPlast unbiased, BB7 unbiased. Attachments 3-6 is the noise situation with only AIDA biased (bPlast was biased but turned off after AIDA was biased.)

10/16 FEEs < 20k noise. Max noise is 250k. Downstream AIDA appears to be seeing additional noise since S100.


11:21 Returning after an hour the noise has increased significantly in many channels. Scale changed to 10000 in histograms, it appears that the problem is from a couple hot channels.

11:41 'Power' cycled AIDA turned off. bPlast turned on, on its own, then BB7. bPlast doesn't seem to notice anything.

12:21 bPlast and BB7 on. AIDA ON. Rates and histograms given by attachment 7-8. More FEEs noisy, but same level as 11:21. There appears to be the same noisy channel in all of the FEEs (?) Some grounding not properly accounted for, or a wire touching somewhere (?)

13:55h: AIDA & BB7 on. Rates and histograms given in attachments 9-10. Same level as bplast & AIDA on. Same noisy channels in FEEs. According to Nic, these correspond to the end of the ribbon cables & is normal.

13:59 AIDA, bPlast & BB7 on before lunch. Same levels of noise as in previous combinations (attachments 11-12).

So it seems we can run with the three sub-systems biased simultaneously.

 

 

Attachment 1: Screenshot_from_2024-06-05_09-36-25.png
Screenshot_from_2024-06-05_09-36-25.png
Attachment 2: Screenshot_from_2024-06-05_09-39-42.png
Screenshot_from_2024-06-05_09-39-42.png
Attachment 3: Screenshot_from_2024-06-05_09-41-44.png
Screenshot_from_2024-06-05_09-41-44.png
Attachment 4: Screenshot_from_2024-06-05_09-41-52.png
Screenshot_from_2024-06-05_09-41-52.png
Attachment 5: Screenshot_from_2024-06-05_09-42-35.png
Screenshot_from_2024-06-05_09-42-35.png
Attachment 6: Screenshot_from_2024-06-05_09-43-43.png
Screenshot_from_2024-06-05_09-43-43.png
Attachment 7: Screenshot_from_2024-06-05_11-40-33.png
Screenshot_from_2024-06-05_11-40-33.png
Attachment 8: Screenshot_from_2024-06-05_11-40-39.png
Screenshot_from_2024-06-05_11-40-39.png
Attachment 9: Screenshot_from_2024-06-05_12-19-14.png
Screenshot_from_2024-06-05_12-19-14.png
Attachment 10: Screenshot_from_2024-06-05_12-20-42.png
Screenshot_from_2024-06-05_12-20-42.png
Attachment 11: Screenshot_from_2024-06-05_13-58-21.png
Screenshot_from_2024-06-05_13-58-21.png
Attachment 12: Screenshot_from_2024-06-05_13-55-08.png
Screenshot_from_2024-06-05_13-55-08.png
Attachment 13: Screenshot_from_2024-06-05_12-20-42.png
Screenshot_from_2024-06-05_12-20-42.png
Attachment 14: Screenshot_from_2024-06-05_12-19-14.png
Screenshot_from_2024-06-05_12-19-14.png
  638   Tue Jun 4 15:40:32 2024 NHTues 4 Jun
DEGAS Array over Snout again

Situation similar to yesterday, most p+n rates good
Attachment 1: Screenshot_from_2024-06-04_16-39-28.png
Screenshot_from_2024-06-04_16-39-28.png
Attachment 2: Screenshot_from_2024-06-04_16-39-47.png
Screenshot_from_2024-06-04_16-39-47.png
  637   Mon Jun 3 14:40:34 2024 JB, NH03 June 2024

15:40 reseated FEE64s adaptors of aida01, aida09, aida13 & aida16.

TEMP OK. HV OK. System wide checks all passed excl. aida07 ADC calibration, see attached.

Histograms and rates look same as previous, reseating aida01 seems to have worked. aida16 still unhappy.

re-reseated aida16. 16:19 - appears to have fixed the problem in the channel.

Pulser test + connected.

bPlast ground removed from frame.

To-do:

- Need clarification about links connected on AIDA n+n FEE64 adaptor boards. Currently, the situation is given by the attachment 8 & 9.

- Fix BB7 grounding, continuity between snout and frame observed.

- Pulser walkthrough for n+n, can do p+n when convienient.

 

 

18.30 Noted FEE64 ADC stats now significantly higher ( attachments 9-10 ) cf. earlier this afternoon (attachments 1-2)

DSSSD bias OFF. FEE64 power OFF.

 

 

Attachment 1: Screenshot_from_2024-06-03_15-49-35.png
Screenshot_from_2024-06-03_15-49-35.png
Attachment 2: Screenshot_from_2024-06-03_15-49-24.png
Screenshot_from_2024-06-03_15-49-24.png
Attachment 3: Screenshot_from_2024-06-03_15-45-54.png
Screenshot_from_2024-06-03_15-45-54.png
Attachment 4: Screenshot_from_2024-06-03_15-42-31.png
Screenshot_from_2024-06-03_15-42-31.png
Attachment 5: Screenshot_from_2024-06-03_15-42-00.png
Screenshot_from_2024-06-03_15-42-00.png
Attachment 6: Screenshot_from_2024-06-03_16-18-49.png
Screenshot_from_2024-06-03_16-18-49.png
Attachment 7: 20240603_172243.jpg
20240603_172243.jpg
Attachment 8: 20240603_172233.jpg
20240603_172233.jpg
Attachment 9: Screenshot_from_2024-06-03_18-34-14.png
Screenshot_from_2024-06-03_18-34-14.png
Attachment 10: Screenshot_from_2024-06-03_18-34-30.png
Screenshot_from_2024-06-03_18-34-30.png
  636   Mon Jun 3 07:35:56 2024 TDMonday 3 June
08.28 DSSSD bias & leakage current - attachments 1-2

      FEE64 temps OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Stat spectra - attachment 5

      per FEE64 1.8.L spectra - attachment 6

      per FEE64 1.8.W spectra - 0us FSR - attachments 7-8

08.39 DAQ stops file S181/R1_36

      FEE64 power OFF
      DSSSD bias ON

11.12 DSSSD bias OFF
Attachment 1: Screenshot_from_2024-06-03_08-27-39.png
Screenshot_from_2024-06-03_08-27-39.png
Attachment 2: Screenshot_from_2024-06-03_08-28-04.png
Screenshot_from_2024-06-03_08-28-04.png
Attachment 3: Screenshot_from_2024-06-03_08-28-37.png
Screenshot_from_2024-06-03_08-28-37.png
Attachment 4: Screenshot_from_2024-06-03_08-29-04.png
Screenshot_from_2024-06-03_08-29-04.png
Attachment 5: Screenshot_from_2024-06-03_08-29-47.png
Screenshot_from_2024-06-03_08-29-47.png
Attachment 6: Screenshot_from_2024-06-03_08-30-52.png
Screenshot_from_2024-06-03_08-30-52.png
Attachment 7: Screenshot_from_2024-06-03_08-34-18.png
Screenshot_from_2024-06-03_08-34-18.png
Attachment 8: Screenshot_from_2024-06-03_08-35-27.png
Screenshot_from_2024-06-03_08-35-27.png
  635   Sun Jun 2 01:45:05 2024 TDSunday 2 June
02.41 DSSSD bias & leakage current - attachment 1

      FEE64 temps OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Stat spectra - attachment 4


06.52 DSSSD bias & leakage current - attachment 5

      FEE64 temps OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

12.13 DSSSD bias & leakage current - attachment 9

      FEE64 temps OK - attachment 10

      ADC data item stats - attachment 11

      per FEE64 Stat spectra - attachment 12


14.51 DSSSD bias & leakage current - attachments 13-14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Stat spectra - attachment 17

21.49 DSSSD bias & leakage current - attachment 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Stat spectra - attachment 21


03.55 DSSSD bias & leakage current - attachment 22

      FEE64 temps OK - attachment 23

      ADC data item stats - attachment 24

      per FEE64 Stat spectra - attachment 25
Attachment 1: Screenshot_from_2024-06-02_02-40-53.png
Screenshot_from_2024-06-02_02-40-53.png
Attachment 2: Screenshot_from_2024-06-02_02-41-25.png
Screenshot_from_2024-06-02_02-41-25.png
Attachment 3: Screenshot_from_2024-06-02_02-41-49.png
Screenshot_from_2024-06-02_02-41-49.png
Attachment 4: Screenshot_from_2024-06-02_02-42-57.png
Screenshot_from_2024-06-02_02-42-57.png
Attachment 5: Screenshot_from_2024-06-02_06-52-56.png
Screenshot_from_2024-06-02_06-52-56.png
Attachment 6: Screenshot_from_2024-06-02_06-53-29.png
Screenshot_from_2024-06-02_06-53-29.png
Attachment 7: Screenshot_from_2024-06-02_06-53-57.png
Screenshot_from_2024-06-02_06-53-57.png
Attachment 8: Screenshot_from_2024-06-02_06-54-52.png
Screenshot_from_2024-06-02_06-54-52.png
Attachment 9: Screenshot_from_2024-06-02_12-13-13.png
Screenshot_from_2024-06-02_12-13-13.png
Attachment 10: Screenshot_from_2024-06-02_12-13-46.png
Screenshot_from_2024-06-02_12-13-46.png
Attachment 11: Screenshot_from_2024-06-02_12-14-07.png
Screenshot_from_2024-06-02_12-14-07.png
Attachment 12: Screenshot_from_2024-06-02_12-15-07.png
Screenshot_from_2024-06-02_12-15-07.png
Attachment 13: Screenshot_from_2024-06-02_14-47-23.png
Screenshot_from_2024-06-02_14-47-23.png
Attachment 14: Screenshot_from_2024-06-02_14-51-12.png
Screenshot_from_2024-06-02_14-51-12.png
Attachment 15: Screenshot_from_2024-06-02_14-51-45.png
Screenshot_from_2024-06-02_14-51-45.png
Attachment 16: Screenshot_from_2024-06-02_14-52-12.png
Screenshot_from_2024-06-02_14-52-12.png
Attachment 17: Screenshot_from_2024-06-02_14-52-46.png
Screenshot_from_2024-06-02_14-52-46.png
Attachment 18: Screenshot_from_2024-06-02_21-47-35.png
Screenshot_from_2024-06-02_21-47-35.png
Attachment 19: Screenshot_from_2024-06-02_21-48-05.png
Screenshot_from_2024-06-02_21-48-05.png
Attachment 20: Screenshot_from_2024-06-02_21-48-30.png
Screenshot_from_2024-06-02_21-48-30.png
Attachment 21: Screenshot_from_2024-06-02_21-49-08.png
Screenshot_from_2024-06-02_21-49-08.png
Attachment 22: Screenshot_from_2024-06-03_03-55-12.png
Screenshot_from_2024-06-03_03-55-12.png
Attachment 23: Screenshot_from_2024-06-03_03-55-56.png
Screenshot_from_2024-06-03_03-55-56.png
Attachment 24: Screenshot_from_2024-06-03_03-56-23.png
Screenshot_from_2024-06-03_03-56-23.png
Attachment 25: Screenshot_from_2024-06-03_03-58-23.png
Screenshot_from_2024-06-03_03-58-23.png
  634   Sat Jun 1 12:48:54 2024 TDSaturday 1 June
13.26 DSSSD bias & leakage current - attachments 1-3

      FEE64 temps OK - attachment 4
       aida02 asic sensor u/s

      All system wide checks OK *except* WR decoder status - attachment 5

      ADC data item stats - attachment 6

      per FEE64 Rate spectra - attachments 7-8

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

      WR timestamps OK - attachment 11

13.59 Data file S181/R1
      alpha background
      BNC PB-5 disconnected?
      slow comparator 0x64

      ADC data item stats - attachment 12
      Merger, TapeServer etc - attachments 13-14

14.02 All histograms & stats zero'd



16.19 DSSSD bias & leakage current - attachment 15

      FEE64 temps OK - attachment 16

      ADC data item stats - attachment 17

      per FEE64 Stat spectra - attachment 18


19.16 DSSSD bias & leakage current - attachments 19-20

      FEE64 temps OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Stat spectra - attachment 23

      2x Check ASIC Control


21.42 DSSSD bias & leakage current - attachment 24

      FEE64 temps OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Stat spectra - attachment 27

00.48 DSSSD bias & leakage current - attachment 28

      FEE64 temps OK - attachment 29

      ADC data item stats - attachment 30

      per FEE64 Stat spectra - attachment 31
Attachment 1: Screenshot_from_2024-06-01_13-26-48.png
Screenshot_from_2024-06-01_13-26-48.png
Attachment 2: Screenshot_from_2024-06-01_13-27-24.png
Screenshot_from_2024-06-01_13-27-24.png
Attachment 3: Screenshot_from_2024-06-01_13-27-47.png
Screenshot_from_2024-06-01_13-27-47.png
Attachment 4: Screenshot_from_2024-06-01_13-40-22.png
Screenshot_from_2024-06-01_13-40-22.png
Attachment 5: Screenshot_from_2024-06-01_13-43-09.png
Screenshot_from_2024-06-01_13-43-09.png
Attachment 6: Screenshot_from_2024-06-01_13-44-41.png
Screenshot_from_2024-06-01_13-44-41.png
Attachment 7: Screenshot_from_2024-06-01_13-45-21.png
Screenshot_from_2024-06-01_13-45-21.png
Attachment 8: Screenshot_from_2024-06-01_13-45-49.png
Screenshot_from_2024-06-01_13-45-49.png
Attachment 9: Screenshot_from_2024-06-01_13-47-30.png
Screenshot_from_2024-06-01_13-47-30.png
Attachment 10: Screenshot_from_2024-06-01_13-48-20.png
Screenshot_from_2024-06-01_13-48-20.png
Attachment 11: Screenshot_from_2024-06-01_13-51-48.png
Screenshot_from_2024-06-01_13-51-48.png
Attachment 12: Screenshot_from_2024-06-01_13-53-14.png
Screenshot_from_2024-06-01_13-53-14.png
Attachment 13: Screenshot_from_2024-06-01_14-00-38.png
Screenshot_from_2024-06-01_14-00-38.png
Attachment 14: Screenshot_from_2024-06-01_14-01-08.png
Screenshot_from_2024-06-01_14-01-08.png
Attachment 15: Screenshot_from_2024-06-01_16-19-23.png
Screenshot_from_2024-06-01_16-19-23.png
Attachment 16: Screenshot_from_2024-06-01_16-27-28.png
Screenshot_from_2024-06-01_16-27-28.png
Attachment 17: Screenshot_from_2024-06-01_16-28-06.png
Screenshot_from_2024-06-01_16-28-06.png
Attachment 18: Screenshot_from_2024-06-01_16-29-07.png
Screenshot_from_2024-06-01_16-29-07.png
Attachment 19: Screenshot_from_2024-06-01_19-18-17.png
Screenshot_from_2024-06-01_19-18-17.png
Attachment 20: Screenshot_from_2024-06-01_19-15-57.png
Screenshot_from_2024-06-01_19-15-57.png
Attachment 21: Screenshot_from_2024-06-01_19-18-48.png
Screenshot_from_2024-06-01_19-18-48.png
Attachment 22: Screenshot_from_2024-06-01_19-19-11.png
Screenshot_from_2024-06-01_19-19-11.png
Attachment 23: Screenshot_from_2024-06-01_19-20-03.png
Screenshot_from_2024-06-01_19-20-03.png
Attachment 24: Screenshot_from_2024-06-01_21-41-48.png
Screenshot_from_2024-06-01_21-41-48.png
Attachment 25: Screenshot_from_2024-06-01_21-42-24.png
Screenshot_from_2024-06-01_21-42-24.png
Attachment 26: Screenshot_from_2024-06-01_21-42-49.png
Screenshot_from_2024-06-01_21-42-49.png
Attachment 27: Screenshot_from_2024-06-01_21-43-54.png
Screenshot_from_2024-06-01_21-43-54.png
Attachment 28: Screenshot_from_2024-06-02_00-48-44.png
Screenshot_from_2024-06-02_00-48-44.png
Attachment 29: Screenshot_from_2024-06-02_00-49-22.png
Screenshot_from_2024-06-02_00-49-22.png
Attachment 30: Screenshot_from_2024-06-02_00-49-48.png
Screenshot_from_2024-06-02_00-49-48.png
Attachment 31: Screenshot_from_2024-06-02_00-50-53.png
Screenshot_from_2024-06-02_00-50-53.png
  633   Mon May 27 17:20:49 2024 TDMonday 27 May
17.50 bPlas cabling, grounds etc connected, power OFF
       CC reports that bPlas power is actually ON

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ASIC settings 2024Apr26-01.29.53
       p+n FFEE64 slow comparator 0xa, n+n Ohmic FEE64 0xf

      System wide checks OK *except* aida02 WR decoder status

      ADC data item stats - attachment 3
       9/16 < 20k, max 234k (aida08)

      per FEE64 Rate spectra - attachments 4-6

      per FEE64 1.8.W spectra - 20us FSR - attachments 7-8

      WR timestamps OK - attachment 9


18.50 FEE64 power OFF
Attachment 1: Screenshot_from_2024-05-27_18-09-02.png
Screenshot_from_2024-05-27_18-09-02.png
Attachment 2: Screenshot_from_2024-05-27_18-09-34.png
Screenshot_from_2024-05-27_18-09-34.png
Attachment 3: Screenshot_from_2024-05-27_18-10-20.png
Screenshot_from_2024-05-27_18-10-20.png
Attachment 4: Screenshot_from_2024-05-27_18-12-21.png
Screenshot_from_2024-05-27_18-12-21.png
Attachment 5: Screenshot_from_2024-05-27_18-13-15.png
Screenshot_from_2024-05-27_18-13-15.png
Attachment 6: Screenshot_from_2024-05-27_18-14-11.png
Screenshot_from_2024-05-27_18-14-11.png
Attachment 7: Screenshot_from_2024-05-27_18-18-40.png
Screenshot_from_2024-05-27_18-18-40.png
Attachment 8: Screenshot_from_2024-05-27_18-19-50.png
Screenshot_from_2024-05-27_18-19-50.png
Attachment 9: Screenshot_from_2024-05-27_18-27-26.png
Screenshot_from_2024-05-27_18-27-26.png
  632   Wed May 22 20:10:19 2024 NH, CC, HMA, MV, ZC, KZAIDA Snout Remounted
AIDA and the Snout have been remounted in S4

I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground

We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections

After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).

It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)

There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already 
Attachment 1: 20240522_153750.jpg
20240522_153750.jpg
Attachment 2: 20240522_153752.jpg
20240522_153752.jpg
Attachment 3: 20240522_153755.jpg
20240522_153755.jpg
Attachment 4: 20240522_153757.jpg
20240522_153757.jpg
Attachment 5: 20240522_153805.jpg
20240522_153805.jpg
Attachment 6: 20240522_153807.jpg
20240522_153807.jpg
Attachment 7: 20240522_153809.jpg
20240522_153809.jpg
Attachment 8: 20240522_153810.jpg
20240522_153810.jpg
Attachment 9: 20240522_153819.jpg
20240522_153819.jpg
  631   Thu May 9 09:08:26 2024 TDCurrent NewMerger install - aida-gsi
[npg@aidas-gsi MIDAS]$ ls -l
total 524
drwxrwxr-x.  2 npg npg    156 Mar 17  2021 Aida
drwxrwxr-x.  2 npg npg   4096 Jul 13  2015 bin_Linux64
drwxrwxr-x.  4 npg npg     38 Jul  3  2018 config
drwxr-xr-x.  3 npg npg     22 Jan 23  2019 config_ORIG
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 Data_Acq
drwxrwxr-x. 25 npg npg   4096 Aug 19  2020 DataPackage
drwxrwxr-x.  3 npg npg     25 Aug 29  2019 DB
drwxr-xr-x.  3 npg npg     25 Jan 23  2019 DB_ORIG
lrwxrwxrwx.  1 npg npg     14 May 30  2019 Embedded -> /home/Embedded
drwxrwxr-x.  2 npg npg   4096 Mar 17  2021 ISOL_Aida
drwxrwxr-x.  2 npg npg   4096 Nov 29  2018 ISOL_Aida_PreApr9
drwxrwxr-x.  7 npg npg     87 Jul  8  2009 lib
drwxrwxr-x.  2 npg npg   4096 Sep 19  2012 lib_Linux64
drwxr-xr-x.  5 npg npg     47 Jan 23  2019 Linux
drwxrwxr-x.  6 npg npg     58 Feb 22  2018 linux-ppc_4xx
drwxr-xr-x.  6 npg npg     58 Jan 23  2019 linux-ppc_4xx_ORIG
drwxrwxr-x.  3 npg npg     73 Feb 16  2022 log
-rwxrw-r--.  1 npg npg  12862 Jul  9  2009 midasicon.ico
-rwxrw-r--.  1 npg npg    773 Jul 23  2012 midasicon.png
-rwxrwxr-x.  1 npg npg    772 Jan 20  2019 MIDASNewGen_BuildAIDA.txt
-rwxrwxr--.  1 npg npg   2532 Jan 19  2019 MIDASNewGen_BuildCore.txt
drwxrwxr-x. 14 npg npg   4096 Feb 22  2018 MIDASpackages
drwxrwxr-x.  6 npg npg    124 May 14  2022 NewMerger
-rw-rw-r--.  1 npg npg 118873 Oct 18  2019 NewMergerdiff.log
-rw-rw-r--.  1 npg npg 118971 Oct 18  2019 NewMergerdiff.log~
drwxrwxr-x.  5 npg npg     52 Jan  8  2019 NewMerger_PreApr9
drwxrwxr-x.  6 npg npg    124 Mar 12  2020 NewMerger_PreMar21
drwxrwxr-x.  7 npg npg    104 Oct 17  2019 NewMerger_PreOct19
-rw-r--r--.  1 npg npg 230517 Mar 12  2020 NewMerger.tgz
drwxr-xr-x. 14 npg npg   4096 Jan 23  2019 OLD
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 SOAP
drwxrwxrwx.  4 npg npg     67 Aug 19  2021 SpecData
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer
drwxr-xr-x.  3 npg npg     21 Jan 23  2019 TapeServer_PreFeb21
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer_PreMar21
drwxr-xr-x.  2 npg npg     44 Feb  2  2021 tcl
drwxr-xr-x. 12 npg npg    152 Jan 23  2019 TclHttpd
drwxrwxr-x. 11 npg npg    155 Oct 26  2017 TclTkpackages
lrwxrwxrwx.  1 npg npg     26 May 30  2019 XilinxLinux -> /home/Embedded/XilinxLinux

[npg@aidas-gsi NewMerger]$ ls -lR
.:
total 32
-rw-rw-r--. 1 npg npg 15096 Apr 16  2019 Aida_ASICreadout.c
drwxrwxr-x. 3 npg npg    23 Oct 18  2019 Html
drwxrwxr-x. 2 npg npg   172 May 14  2022 include
drwxrwxr-x. 8 npg npg   256 May 14  2022 MergeServer
-rw-rw-r--. 1 npg npg 11048 Aug 29  2019 R3B_readout.c
-rw-rw-r--. 1 npg npg   152 Oct 17  2019 README
drwxrwxr-x. 2 npg npg    34 Apr 29  2021 tcl

./Html:
total 0
drwxrwxr-x. 6 npg npg 187 Oct 13  2020 NewMerger

./Html/NewMerger:
total 60
lrwxrwxrwx. 1 npg npg    21 Oct 18  2019 Html -> /MIDAS/NewMerger/Html
drwxrwxr-x. 2 npg npg     6 Aug 26  2019 LinkRates
-rw-rw-r--. 1 npg npg  1210 Aug 26  2019 NewMerger.js
-rw-rw-r--. 1 npg npg  9417 Aug 29  2019 NewMerger.tcl
-rw-rw-r--. 1 npg npg  9332 Aug 26  2019 NewMerger.tcl~
-rw-rw-r--. 1 npg npg 13052 Aug 27  2019 NewMerger.tml
-rw-rw-r--. 1 npg npg 13851 Aug 26  2019 NewMerger.tml~
drwxrwxr-x. 2 npg npg    82 Oct 13  2020 Options
drwxrwxr-x. 2 npg npg   102 Oct 13  2020 RunControl
drwxrwxr-x. 2 npg npg   192 Aug 29  2019 Stats

./Html/NewMerger/LinkRates:
total 0

./Html/NewMerger/Options:
total 24
-rwxrwxr-x. 1 npg npg  725 Aug 26  2019 Options.js
-rwxrwxr-x. 1 npg npg 3934 Aug 26  2019 Options.tcl
-rwxrwxr-x. 1 npg npg 7864 Aug 27  2019 Options.tml
-rw-rw-r--. 1 npg npg 7864 Aug 27  2019 Options.tml~

./Html/NewMerger/RunControl:
total 44
-rw-rw-r--. 1 npg npg  4791 Aug 26  2019 implementation.tcl
-rw-rw-r--. 1 npg npg  1643 Aug 26  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg   539 Aug 26  2019 stats.tcl
-rw-rw-r--. 1 npg npg 12328 Aug 29  2019 sys.tcl
-rw-rw-r--. 1 npg npg 12231 Aug 26  2019 sys.tcl~

./Html/NewMerger/Stats:
total 52
-rw-rw-r--. 1 npg npg 2880 Aug 26  2019 MergeStatsALL.tcl
-rw-rw-r--. 1 npg npg 1013 Aug 26  2019 MergeStats.js
-rw-rw-r--. 1 npg npg 1534 Aug 26  2019 MergeStatsSINGLE.tcl
-rw-rw-r--. 1 npg npg 4259 Aug 26  2019 MergeStats.tcl
-rw-rw-r--. 1 npg npg 8467 Aug 27  2019 MergeStats.tml
-rw-rw-r--. 1 npg npg 8338 Aug 26  2019 MergeStats.tml~
-rw-rw-r--. 1 npg npg 1829 Aug 29  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg 1943 Aug 29  2019 stats.defn.tcl~

./include:
total 52
-rw-rw-r--. 1 npg npg 1079 Apr 16  2019 buffer.h
-rw-rw-r--. 1 npg npg 4473 Apr 16  2019 common.h
-rw-rw-r--. 1 npg npg 1199 Jul 16  2019 merge.h
-rw-rw-r--. 1 npg npg 1273 Apr 16  2019 message.h
-rw-rw-r--. 1 npg npg 1273 Oct 17  2019 message.h~
-rw-rw-r--. 1 npg npg 5288 Aug 29  2019 netvar.h
-rw-rw-r--. 1 npg npg 5256 Apr 16  2019 netvar.h~
-rw-rw-r--. 1 npg npg  912 Apr 16  2019 services.h
-rw-rw-r--. 1 npg npg  886 Apr 18  2019 stats.h
-rw-rw-r--. 1 npg npg 2710 Apr 16  2019 transfer.h

./MergeServer:
total 60
-rw-rw-r--. 1 npg npg  2606 Apr 16  2019 alloc_data_area.c
drwxr-xr-x. 2 npg npg    69 Aug 27  2019 bin64
drwxrwxr-x. 2 npg npg   128 Apr  6  2021 bin64.BackUp
-rw-rw-r--. 1 npg npg 10510 Sep 19  2019 buffer.c
-rwxrwxr-x. 1 npg npg   305 Apr 16  2019 build
drwxrwxr-x. 3 npg npg   279 May 14  2022 link
-rw-rw-r--. 1 npg npg   494 Apr 16  2019 Makefile
drwxrwxr-x. 3 npg npg   160 Apr  6  2021 master
drwxrwxr-x. 3 npg npg   143 Apr  6  2021 merge
-rw-rw-r--. 1 npg npg  3225 Sep 19  2019 message.c
-rw-rw-r--. 1 npg npg  3199 Sep 19  2019 netvar.c
-rwxrwxr-x. 1 npg npg   397 Aug 26  2019 run
-rw-rw-r--. 1 npg npg 15309 Sep 19  2019 signal.c
drwxrwxr-x. 3 npg npg   101 Sep 19  2019 statrate
-rw-rw-r--. 1 npg npg  1548 Sep 19  2019 stats.c
-rw-rw-r--. 1 npg npg   593 Apr 16  2019 version.c

./MergeServer/bin64:
total 148
-rwxr-xr-x. 1 npg npg 51964 May 11  2021 link64
-rwxr-xr-x. 1 npg npg 33312 May 11  2021 master64
-rwxr-xr-x. 1 npg npg 36027 May 11  2021 merge64
-rwxr-xr-x. 1 npg npg 21370 May 11  2021 statrate64

./MergeServer/bin64.BackUp:
total 256
-rwxr-xr-x. 1 npg npg 51948 Mar 16  2021 link64
-rwxrwxr-x. 1 npg npg 33184 Apr  6  2021 master64
-rwxr-xr-x. 1 npg npg 33312 Mar 16  2021 master64.new
-rwxr-xr-x. 1 npg npg 36027 Mar 16  2021 merge64
-rwxr-xr-x. 1 npg npg 35368 Mar  9  2021 merge64.new
-rwxrwxr-x. 1 npg npg 35910 Mar  8  2021 merge64.orig
-rwxr-xr-x. 1 npg npg 21370 Mar 16  2021 statrate64

./MergeServer/link:
total 164
-rw-rw-r--. 1 npg npg 10216 Oct 18  2019 handle_buffer.c
-rw-rw-r--. 1 npg npg 24623 Sep 19  2019 link_server.c
-rw-rw-r--. 1 npg npg 24732 Oct 18  2019 link_server.c~
drwxrwxr-x. 2 npg npg   217 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg  7051 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg  7029 Oct 18  2019 main.c~
-rw-rw-r--. 1 npg npg 14464 Sep 19  2019 processAIDADATAbuffer.c
-rw-rw-r--. 1 npg npg 14463 Oct 18  2019 processAIDADATAbuffer.c~
-rw-rw-r--. 1 npg npg 14950 Sep 19  2019 processR3BDATAbuffer.c
-rw-rw-r--. 1 npg npg 14925 Oct 18  2019 processR3BDATAbuffer.c~
-rw-rw-r--. 1 npg npg 15077 Aug 22  2019 processR3BDATAbuffer.c_PreAug22

./MergeServer/link/Linux64:
total 176
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  6216 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 29840 Oct 24  2019 handle_buffer.o
-rwxrwxr-x. 1 npg npg 50660 Oct 24  2019 link64
-rw-rw-r--. 1 npg npg 24464 Oct 24  2019 link_server.o
-rw-rw-r--. 1 npg npg 14864 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   795 Aug 22  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12848 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/master:
total 40
-rw-rw-r--. 1 npg npg 7850 Sep 18  2019 handle_signals.c
drwxrwxr-x. 2 npg npg  223 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 6952 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 6973 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 1722 Aug 29  2019 start_link.c
-rw-rw-r--. 1 npg npg 1885 Aug 22  2019 start_link.c~
-rw-rw-r--. 1 npg npg 1372 Apr 16  2019 start_merge.c
-rw-rw-r--. 1 npg npg 1345 Apr 16  2019 start_statrate.c

./MergeServer/master/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  4168 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 22840 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   793 Jul 16  2019 Makefile
-rwxrwxr-x. 1 npg npg 33184 Oct 24  2019 master64
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 14616 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 start_link.o
-rw-rw-r--. 1 npg npg  2904 Oct 24  2019 start_merge.o
-rw-rw-r--. 1 npg npg  2808 Oct 24  2019 start_statrate.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/merge:
total 48
drwxrwxr-x. 2 npg npg  228 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 5130 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 4579 Oct 24  2019 mergeAIDA.c
-rw-rw-r--. 1 npg npg 4119 Oct 17  2019 #merge.c#
-rw-rw-r--. 1 npg npg 4148 Oct 21  2019 merge.c
-rw-rw-r--. 1 npg npg 3703 Sep 19  2019 mergeR3B.c
-rw-rw-r--. 1 npg npg 8060 Sep 17  2019 merge_server.c
-rw-rw-r--. 1 npg npg 2394 Sep 17  2019 storage.c

./MergeServer/merge/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  3872 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 10944 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   789 Jul 16  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rwxrwxr-x. 1 npg npg 35910 Oct 24  2019 merge64
-rw-rw-r--. 1 npg npg  9064 Oct 24  2019 merge.o
-rw-rw-r--. 1 npg npg  8160 Oct 24  2019 merge_server.o
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12744 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  4048 Oct 24  2019 storage.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/statrate:
total 16
drwxrwxr-x. 2 npg npg  161 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 1876 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 1717 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c~

./MergeServer/statrate/Linux64:
total 80
-rw-rw-r--. 1 npg npg  8880 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   725 Apr 16  2019 Makefile
-rw-rw-r--. 1 npg npg   728 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5376 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 10144 Oct 24  2019 signal.o
-rwxrwxr-x. 1 npg npg 21370 Oct 24  2019 statrate64
-rw-rw-r--. 1 npg npg  4664 Oct 24  2019 statrate_server.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1872 Oct 24  2019 version.o

./tcl:
total 4
-rw-rw-r--. 1 npg npg 1775 Aug 26  2019 NewMergerControl.tcl
  630   Tue May 7 13:25:00 2024 JB, CC, NH, HMATuesday 6 May - dismount snout

Status before dismount

14:26 DSSSD#1 119.99 V @ 17.968 uA

          DSSSD#2 119.95 V @ 11.212 uA

          Leakage currents slightly elevated from Monday.

S4 temperature at 26 °C.

14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.

Plan:

- Dismount snout.

- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.

 

 

Attachment 1: Screenshot_from_2024-05-07_14-25-52.png
Screenshot_from_2024-05-07_14-25-52.png
Attachment 2: Screenshot_from_2024-05-07_14-29-13.png
Screenshot_from_2024-05-07_14-29-13.png
  629   Mon May 6 07:10:26 2024 TDMonday 6 May
08.11 DSSSD#1 bias -120V leakage current -17.0uA
      DSSSD#2 bias -120V leakage current -10.5uA
      S4 temperature 25 deg C

      cf. 04.00 30.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.9uA
      DSSSD#2 -13.0uA

      Leakage currents currently c. 80% of 30.4.24 values at same temperature 
Attachment 1: Capture.PNG
Capture.PNG
  628   Sun May 5 10:22:24 2024 TDSunday 5 May
11.22 DSSSD#1 bias -120V leakage current -17.1uA
      DSSSD#2 bias -120V leakage current -10.6uA
      S4 temperature 25 deg C

      cf. 07.00 29.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.8uA
      DSSSD#2 -12.8uA

      Leakage currents currently c. 80% of 29.4.24 values at same temperature 
Attachment 1: Capture.PNG
Capture.PNG
  627   Sat May 4 07:36:07 2024 TDSaturday 4 May
08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
      FEE64 temperatures OK - attachment 3
      ADC, correlation scalers data item stats OK attachments 4-5
           11/16 <20k max aida16 174k
      per FEE64 Rate spectra - attachmnt 6


09.44 per p+n FEE64 1.8.L spectra - attachment 7
       aida09 pulser peak width 58 ch FWHM
      per FEE64 1.8.W spectra - attachments 8-10

09.47 bias cycle - off and then on
      no significant change in leakage current

09.57 FEE64 power OFF
      DSSSD bias ON
Attachment 1: Screenshot_from_2024-05-04_08-35-38.png
Screenshot_from_2024-05-04_08-35-38.png
Attachment 2: Screenshot_from_2024-05-04_08-33-29.png
Screenshot_from_2024-05-04_08-33-29.png
Attachment 3: Screenshot_from_2024-05-04_08-31-09.png
Screenshot_from_2024-05-04_08-31-09.png
Attachment 4: Screenshot_from_2024-05-04_08-31-45.png
Screenshot_from_2024-05-04_08-31-45.png
Attachment 5: Screenshot_from_2024-05-04_08-32-21.png
Screenshot_from_2024-05-04_08-32-21.png
Attachment 6: Screenshot_from_2024-05-04_08-32-41.png
Screenshot_from_2024-05-04_08-32-41.png
Attachment 7: Screenshot_from_2024-05-04_09-37-27.png
Screenshot_from_2024-05-04_09-37-27.png
Attachment 8: Screenshot_from_2024-05-04_09-42-27.png
Screenshot_from_2024-05-04_09-42-27.png
Attachment 9: Screenshot_from_2024-05-04_09-42-56.png
Screenshot_from_2024-05-04_09-42-56.png
Attachment 10: Screenshot_from_2024-05-04_09-44-31.png
Screenshot_from_2024-05-04_09-44-31.png
Attachment 11: Capture.PNG
Capture.PNG
  626   Thu May 2 23:08:43 2024 TDFriday 3 May
00.05 FEE64 temperatures OK
      ADC data item stats OK

09.31 FEE64 temperatures OK
      ADC data item stats OK

13.15 FEE64 temperatures OK
      ADC data item stats OK
       9/16 <20k max aida16 164k

20.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida03 no data - DAQ reset
       9/16 <20k max aida16 158k

23.15 FEE64 temperatures OK
      ADC data item stats
       aida03 no data - DAQ reset
       10/16 <20k max aida16 153k
  625   Thu May 2 00:16:22 2024 TD, NHThursday 2 May
01.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       6/16 < 20k


10.25 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 restarted
       7/16 < 20k, max 220k aida16

      all histograms zero'd

13.20 per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 59 ch FWHM
      FEE64 temperatures OK
      ADC data item stats
       aida04 no data
       5/16 < 20k, max 332k aida08 

14.37
  Moved Time Machine delayed scaler from aida04 to aida07
  aida04 no data
  Reset merger etc 
  aida05 no data
  Reset merger etc 
  All FEEs with data again

  S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C

17.26 FEE64 temperatures OK
      ADC data item stats OK

19.28 FEE64 temperatures OK
      ADC data item stats 
       aida02 no data - DAQ reset
      
Attachment 1: Screenshot_from_2024-05-02_13-21-07.png
Screenshot_from_2024-05-02_13-21-07.png
Attachment 2: Screenshot_from_2024-05-02_13-22-16.png
Screenshot_from_2024-05-02_13-22-16.png
Attachment 3: Screenshot_from_2024-05-02_13-22-43.png
Screenshot_from_2024-05-02_13-22-43.png
  624   Wed May 1 08:59:15 2024 TDWednesday 1 May
04.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida04 no data
      TapeServer no storage mode

09.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida02 & aida04 no data
      DSSSD bias and leakage current - Grafana - attachment 1

19.28 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       noted increase in rates for aid08 and aida16
       5/16 < 20k
Attachment 1: Capture.PNG
Capture.PNG
  623   Tue Apr 30 09:06:45 2024 TD, NHTuesday 30 April
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4
       9/16 FEE64s < 20k 

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachments 6-7
       aida02, aida04 inactive

11.20 DAQ restarted
      aida02 and aida04 now producing data


11.30 analysis data file R22_313
      all deadtimes <1%
      no timewarps
      HEC data item rate 560Hz (all from one channel of aida08 asic #3)
      aida03 & aida04 MBS rates c. 24kHz


14.32 FEE64 temperatures OK
      ADC data item stats OK
       all FEE64s producing data

14.43 all histograms & stats zero'd

[NH checks here]

21.26 aida04 stopped forwarding to merger
 the "quick fix" (stop merger, rerun) worked

MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability

System Wide Checks
 clocks PASSED
 ADC IGNORED
 WR:
		 Base 		Current 	Difference
aida02 fault 	 0xe4d8 : 	 0x65c0 : 	 -32536  
aida03 fault 	 0x2371 : 	 0x2380 : 	 15  
aida04 fault 	 0xa210 : 	 0xaf58 : 	 3400  
aida06 fault 	 0x3624 : 	 0x3626 : 	 2  
White Rabbit error counter test result: Passed 12, Failed 4

FPGA: 
	
			 Base 		Current 		Difference
aida02 fault 	 0xaa3b : 	 0xc455 : 	 6682  
aida04 fault 	 0x5a4 : 	 0x6ba : 	 278  
aida06 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1

Baselines reset

22.50 FEE64 temps ok
      merger ok
       all links active
      ADC data item stats ok
Attachment 1: Screenshot_from_2024-04-30_10-05-08.png
Screenshot_from_2024-04-30_10-05-08.png
Attachment 2: Screenshot_from_2024-04-30_10-05-37.png
Screenshot_from_2024-04-30_10-05-37.png
Attachment 3: Screenshot_from_2024-04-30_10-01-32.png
Screenshot_from_2024-04-30_10-01-32.png
Attachment 4: Screenshot_from_2024-04-30_10-02-03.png
Screenshot_from_2024-04-30_10-02-03.png
Attachment 5: Screenshot_from_2024-04-30_10-03-26.png
Screenshot_from_2024-04-30_10-03-26.png
Attachment 6: Screenshot_from_2024-04-30_10-03-58.png
Screenshot_from_2024-04-30_10-03-58.png
Attachment 7: R22_313
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  246802750 (  787077.4 Hz)
        Other data format:   15117251 (   48210.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        802 (       2.6 Hz)
                               RESUME:        802 (       2.6 Hz)
                              SYNC100:      33168 (     105.8 Hz)
                              WR48-63:      33168 (     105.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   15049311 (   47993.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     182597 (     582.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.569 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.410                0.000
  4                0.000                0.000
  5                0.208                0.000
  6                0.000                0.000
  7                1.577                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                1.890                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.784                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4403022      10517          0          0          0          0        556        556          0       9405          0          0
  1    4797622      11663          0          0          0          0        607        607          0      10449          0          0
  2    9960742    7551016          0          0          0          0       2246       2246          0    7546524          0          0
  3   26989916    7491649          0          0         64         64       4294       4294          0    7482933          0          0
  4    2737535        758          0          0          0          0        379        379          0          0          0          0
  5   21805196       5662          0          0         70         70       2761       2761          0          0          0          0
  6    6978804       1886          0          0          0          0        943        943          0          0          0          0
  7   53329550      13820          0          0        126        126       6784       6784          0          0          0     182597
  8     479454        122          0          0          0          0         61         61          0          0          0          0
  9    5592721       1538          0          0          0          0        769        769          0          0          0          0
 10   39929583      11138          0          0        441        441       5128       5128          0          0          0          0
 11    2351324        636          0          0          0          0        318        318          0          0          0          0
 12    5309169       1226          0          0          0          0        613        613          0          0          0          0
 13    3913898       1006          0          0          0          0        503        503          0          0          0          0
 14    2364855        578          0          0          0          0        289        289          0          0          0          0
 15   55859359      14036          0          0        101        101       6917       6917          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
  622   Mon Apr 29 08:11:16 2024 TDMonday 29 April
08.52 Beam off

09.11 New data file R22
      Monitor accumulated implanted activity in bPlas and DSSSDs

      ASIC settings 2024Apr26-01.29.53

      DSSSD bias & leakage current OK - attachments 1-3

      FEE64 temperatures OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      Merger etc - attachment 7

      analysis data file R22_19 - attachment 8
       all deadtimes 0.3%, or less
       no timewarps
       HEC data item rate 390Hz

   
10.00 all histograms & stats zero'd


13.24 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temperatures OK - attachment 11

      ADC data item stats - attachment 12

      per FEE64 Rate spectra - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-16
       common x/y axes - peak height proxy for width
       aida09 pulser peak width 58 ch FWHM (~41keV FWHM)  cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
       electronic noise today very similar to electronic noise Monday 22 April

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-19

      Merger etc - attachment 20

      analysis data file R22_60 - attachment 21
       max deadtime 1.7% (aida11), all others <1% 
       no timewarps
       HEC data item rate 710Hz

14.13 ucesb - attachment 22

14.15 aida04 not producing data

14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618

15.27 *all* FEE64s stop producing data - reason unknown

16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618


16.29 DSSSD bias & leakage current OK - attachments 23-24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27

      Merger etc - attachment 28


20.45 DSSSD bias & leakage current OK - attachments 29-30

      FEE64 temperatures OK - attachment 32

      ADC data item stats - attachment 31

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

20.54  analysis data file R22_145 - attachment 35
       max deadtime 1.5% (aida11), all others <1% 
       no timewarps
       HEC data item rate 775Hz



03.45 DSSSD bias & leakage current OK - attachments 36-37

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41
Attachment 1: Screenshot_from_2024-04-29_09-13-12.png
Screenshot_from_2024-04-29_09-13-12.png
Attachment 2: Screenshot_from_2024-04-29_09-13-37.png
Screenshot_from_2024-04-29_09-13-37.png
Attachment 3: Screenshot_from_2024-04-29_09-13-56.png
Screenshot_from_2024-04-29_09-13-56.png
Attachment 4: Screenshot_from_2024-04-29_09-14-31.png
Screenshot_from_2024-04-29_09-14-31.png
Attachment 5: Screenshot_from_2024-04-29_09-15-05.png
Screenshot_from_2024-04-29_09-15-05.png
Attachment 6: Screenshot_from_2024-04-29_09-15-48.png
Screenshot_from_2024-04-29_09-15-48.png
Attachment 7: Screenshot_from_2024-04-29_09-17-25.png
Screenshot_from_2024-04-29_09-17-25.png
Attachment 8: R22_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  245162657 (  661369.3 Hz)
        Other data format:   16757344 (   45205.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        432 (       1.2 Hz)
                               RESUME:        432 (       1.2 Hz)
                              SYNC100:      33381 (      90.1 Hz)
                              WR48-63:      33381 (      90.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   16689718 (   45023.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     145285 (     391.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      370.689 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.421                0.000
  4                0.000                0.000
  5                0.062                0.000
  6                0.000                0.000
  7                1.097                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                1.032                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.195                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4689453      12333          0          0          0          0        606        606          0      11121          0        151
  1    6714412    1102163          0          0          0          0        995        995          0    1100173          0        247
  2    9093299    6196435          0          0          0          0       2024       2024          0    6192387          0        151
  3   31978220    6022790          0          0         67         67       4731       4731          0    6013194          0        212
  4    3345145    1695016          0          0          0          0        641        641          0    1693734          0         75
  5   18560198    1684371          0          0         12         12       2619       2619          0    1679109          0        279
  6    7471590       2034          0          0          0          0       1017       1017          0          0          0        150
  7   51047482      13240          0          0        105        105       6515       6515          0          0          0     143158
  8    1099097        280          0          0          0          0        140        140          0          0          0        152
  9    7056038       1776          0          0          0          0        888        888          0          0          0        136
 10   35026945       9268          0          0        144        144       4490       4490          0          0          0         97
 11    2488733        676          0          0          0          0        338        338          0          0          0        118
 12    6088987       1632          0          0          0          0        816        816          0          0          0         53
 13    5057582       1278          0          0          0          0        639        639          0          0          0        108
 14    2541330        642          0          0          0          0        321        321          0          0          0        127
 15   52904146      13410          0          0        104        104       6601       6601          0          0          0         71
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.137s (  965.696 blocks/s,  60.356 Mb/s)
Attachment 9: Screenshot_from_2024-04-29_13-24-45.png
Screenshot_from_2024-04-29_13-24-45.png
Attachment 10: Screenshot_from_2024-04-29_13-25-48.png
Screenshot_from_2024-04-29_13-25-48.png
Attachment 11: Screenshot_from_2024-04-29_13-26-19.png
Screenshot_from_2024-04-29_13-26-19.png
Attachment 12: Screenshot_from_2024-04-29_13-26-51.png
Screenshot_from_2024-04-29_13-26-51.png
Attachment 13: Screenshot_from_2024-04-29_13-27-33.png
Screenshot_from_2024-04-29_13-27-33.png
Attachment 14: Screenshot_from_2024-04-29_13-28-45.png
Screenshot_from_2024-04-29_13-28-45.png
Attachment 15: Screenshot_from_2024-04-29_13-29-15.png
Screenshot_from_2024-04-29_13-29-15.png
Attachment 16: Screenshot_from_2024-04-29_13-30-17.png
Screenshot_from_2024-04-29_13-30-17.png
Attachment 17: Screenshot_from_2024-04-29_13-32-51.png
Screenshot_from_2024-04-29_13-32-51.png
Attachment 18: Screenshot_from_2024-04-29_13-33-20.png
Screenshot_from_2024-04-29_13-33-20.png
Attachment 19: Screenshot_from_2024-04-29_13-34-07.png
Screenshot_from_2024-04-29_13-34-07.png
Attachment 20: Screenshot_from_2024-04-29_13-34-32.png
Screenshot_from_2024-04-29_13-34-32.png
Attachment 21: R22_60
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258414198 (  779475.1 Hz)
        Other data format:    3505804 (   10574.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        880 (       2.7 Hz)
                               RESUME:        880 (       2.7 Hz)
                              SYNC100:      33235 (     100.2 Hz)
                              WR48-63:      33235 (     100.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3437574 (   10369.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     235960 (     711.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      331.523 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.095                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.217                0.000
  6                0.140                0.000
  7                2.432                0.000
  8                0.000                0.000
  9                0.011                0.000
 10                5.658                0.000
 11                0.000                0.000
 12                0.051                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                2.652                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4927110      11209          0          0          4          4        628        628          0       9945          0         39
  1    5556527     109296          0          0          0          0        741        741          0     107814          0         59
  2   10335604    3081300          0          0          0          0       1668       1668          0    3077964          0         29
  3          0          0          0          0          0          0          0          0          0          0          0          0
  4    3176904     121984          0          0          0          0        428        428          0     121128          0         19
  5   28235398     128309          0          0         73         73       3720       3720          0     120723          0         52
  6    7534053       1958          0          0          2          2        977        977          0          0          0         22
  7   65092755      17088          0          0        173        173       8371       8371          0          0          0     235567
  8     631770        138          0          0          0          0         69         69          0          0          0         28
  9    6806363       1686          0          0          1          1        842        842          0          0          0         28
 10   44391302      12266          0          0        505        505       5628       5628          0          0          0         16
 11    2632024        670          0          0          0          0        335        335          0          0          0         21
 12    5950079       1370          0          0          1          1        684        684          0          0          0         14
 13    4496081       1136          0          0          0          0        568        568          0          0          0         29
 14    2644608        652          0          0          0          0        326        326          0          0          0         25
 15   66003620      16742          0          0        121        121       8250       8250          0          0          0         12
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   37.793s (  846.718 blocks/s,  52.920 Mb/s)
Attachment 22: Capture.PNG
Capture.PNG
Attachment 23: Screenshot_from_2024-04-29_16-29-34.png
Screenshot_from_2024-04-29_16-29-34.png
Attachment 24: Screenshot_from_2024-04-29_16-29-54.png
Screenshot_from_2024-04-29_16-29-54.png
Attachment 25: Screenshot_from_2024-04-29_16-30-39.png
Screenshot_from_2024-04-29_16-30-39.png
Attachment 26: Screenshot_from_2024-04-29_16-31-01.png
Screenshot_from_2024-04-29_16-31-01.png
Attachment 27: Screenshot_from_2024-04-29_16-31-51.png
Screenshot_from_2024-04-29_16-31-51.png
Attachment 28: Screenshot_from_2024-04-29_16-32-12.png
Screenshot_from_2024-04-29_16-32-12.png
Attachment 29: Screenshot_from_2024-04-29_20-45-26.png
Screenshot_from_2024-04-29_20-45-26.png
Attachment 30: Screenshot_from_2024-04-29_20-46-25.png
Screenshot_from_2024-04-29_20-46-25.png
Attachment 31: Screenshot_from_2024-04-29_20-46-52.png
Screenshot_from_2024-04-29_20-46-52.png
Attachment 32: Screenshot_from_2024-04-29_20-47-20.png
Screenshot_from_2024-04-29_20-47-20.png
Attachment 33: Screenshot_from_2024-04-29_20-48-08.png
Screenshot_from_2024-04-29_20-48-08.png
Attachment 34: Screenshot_from_2024-04-29_20-48-56.png
Screenshot_from_2024-04-29_20-48-56.png
Attachment 35: R22_145
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  247656598 (  992529.4 Hz)
        Other data format:   14263402 (   57163.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        881 (       3.5 Hz)
                               RESUME:        881 (       3.5 Hz)
                              SYNC100:      32930 (     132.0 Hz)
                              WR48-63:      32930 (     132.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   14195780 (   56892.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     193328 (     774.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      249.521 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.003                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.318                0.000
  4                0.000                0.000
  5                1.537                0.000
  6                0.000                0.000
  7                1.906                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                3.841                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.886                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6683029       9310          0          0          5          5        906        906          0       7488          0         74
  1    5367157    1313019          0          0          0          0        827        827          0    1311365          0         63
  2   11584446    4358913          0          0          0          0       1980       1980          0    4354953          0         37
  3   22834625    4167004          0          0         37         37       3370       3370          0    4160190          0         57
  4    3277605    2130735          0          0          0          0        657        657          0    2129421          0         19
  5   24294581    2239601          0          0        194        194       3425       3425          0    2232363          0         49
  6    6724993       1760          0          0          0          0        880        880          0          0          0         28
  7   50507044      12910          0          0        126        126       6329       6329          0          0          0     192806
  8     656116        188          0          0          0          0         94         94          0          0          0         10
  9    5715462       1410          0          0          0          0        705        705          0          0          0         12
 10   39787841      10690          0          0        431        431       4914       4914          0          0          0          4
 11    3128950        784          0          0          0          0        392        392          0          0          0         67
 12    4910404       1244          0          0          0          0        622        622          0          0          0         13
 13    3786534        940          0          0          0          0        470        470          0          0          0         59
 14    2967123        782          0          0          0          0        391        391          0          0          0          2
 15   55430688      14112          0          0         88         88       6968       6968          0          0          0         28
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.727s (  948.807 blocks/s,  59.300 Mb/s)
Attachment 36: Screenshot_from_2024-04-30_03-45-13.png
Screenshot_from_2024-04-30_03-45-13.png
Attachment 37: Screenshot_from_2024-04-30_03-46-07.png
Screenshot_from_2024-04-30_03-46-07.png
Attachment 38: Screenshot_from_2024-04-30_03-46-43.png
Screenshot_from_2024-04-30_03-46-43.png
Attachment 39: Screenshot_from_2024-04-30_03-47-10.png
Screenshot_from_2024-04-30_03-47-10.png
Attachment 40: Screenshot_from_2024-04-30_03-47-40.png
Screenshot_from_2024-04-30_03-47-40.png
Attachment 41: Screenshot_from_2024-04-30_03-48-26.png
Screenshot_from_2024-04-30_03-48-26.png
  621   Sun Apr 28 22:52:06 2024 Marc0:00-08:00 Monday 29 April

Just another manic magic Monday, ....

Taking over Betool's AIDA shift. All is good.

This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.

01:15: AIDA04 is down again. DAQ shift crew informed.

01:27: AIDA04 is back and shift crew has been informed.

01:30 Checks:

     Bias and leakage current - Attachment 1-2

     Temperatures - Attachment 3

     ADC Data item - Attachment 4

     Rate Spectra - Attachment 5

     UCESB - Attachment  6

     Timestamp checks - Attachement 7

06:00 It has run very smoothly.   

 

Attachment 1: Screenshot_from_2024-04-29_01-33-27.png
Screenshot_from_2024-04-29_01-33-27.png
Attachment 2: Screenshot_from_2024-04-29_01-34-37.png
Screenshot_from_2024-04-29_01-34-37.png
Attachment 3: Screenshot_from_2024-04-29_01-36-12.png
Screenshot_from_2024-04-29_01-36-12.png
Attachment 4: Screenshot_from_2024-04-29_01-36-41.png
Screenshot_from_2024-04-29_01-36-41.png
Attachment 5: Screenshot_from_2024-04-29_01-37-56.png
Screenshot_from_2024-04-29_01-37-56.png
Attachment 6: Screenshot_from_2024-04-29_01-32-39.png
Screenshot_from_2024-04-29_01-32-39.png
Attachment 7: Screenshot_from_2024-04-29_01-42-58.png
Screenshot_from_2024-04-29_01-42-58.png
  620   Sun Apr 28 20:35:57 2024 Betool Alayed16:0-00:00 28 April 2024

21:30 screenshots

and

23:24 screenshots

Attachment 1: Screenshot_from_2024-04-28_21-30-19.png
Screenshot_from_2024-04-28_21-30-19.png
Attachment 2: Screenshot_from_2024-04-28_21-31-11.png
Screenshot_from_2024-04-28_21-31-11.png
Attachment 3: Screenshot_from_2024-04-28_21-32-07.png
Screenshot_from_2024-04-28_21-32-07.png
Attachment 4: Screenshot_from_2024-04-28_21-32-51.png
Screenshot_from_2024-04-28_21-32-51.png
Attachment 5: Screenshot_from_2024-04-28_21-35-10.png
Screenshot_from_2024-04-28_21-35-10.png
Attachment 6: Screenshot_from_2024-04-28_23-35-25.png
Screenshot_from_2024-04-28_23-35-25.png
Attachment 7: Screenshot_from_2024-04-28_23-34-31.png
Screenshot_from_2024-04-28_23-34-31.png
Attachment 8: Screenshot_from_2024-04-28_23-33-42.png
Screenshot_from_2024-04-28_23-33-42.png
Attachment 9: Screenshot_from_2024-04-28_23-32-28.png
Screenshot_from_2024-04-28_23-32-28.png
Attachment 10: Screenshot_from_2024-04-28_23-31-51.png
Screenshot_from_2024-04-28_23-31-51.png
  619   Sun Apr 28 16:26:47 2024 Betool Alayed16:00-00:00 Sunday 28 April

19:24 pm screenshots

and

17:18 pm screenshots

Attachment 1: Screenshot_from_2024-04-28_17-18-38.png
Screenshot_from_2024-04-28_17-18-38.png
Attachment 2: Screenshot_from_2024-04-28_17-13-13.png
Screenshot_from_2024-04-28_17-13-13.png
Attachment 3: Screenshot_from_2024-04-28_17-11-48.png
Screenshot_from_2024-04-28_17-11-48.png
Attachment 4: Screenshot_from_2024-04-28_17-10-32.png
Screenshot_from_2024-04-28_17-10-32.png
Attachment 5: Screenshot_from_2024-04-28_17-04-10.png
Screenshot_from_2024-04-28_17-04-10.png
Attachment 6: Screenshot_from_2024-04-28_19-18-42.png
Screenshot_from_2024-04-28_19-18-42.png
Attachment 7: Screenshot_from_2024-04-28_19-20-10.png
Screenshot_from_2024-04-28_19-20-10.png
Attachment 8: Screenshot_from_2024-04-28_19-22-19.png
Screenshot_from_2024-04-28_19-22-19.png
Attachment 9: Screenshot_from_2024-04-28_19-23-19.png
Screenshot_from_2024-04-28_19-23-19.png
Attachment 10: Screenshot_from_2024-04-28_19-24-40.png
Screenshot_from_2024-04-28_19-24-40.png
  618   Sun Apr 28 16:07:59 2024 TD[How To] Recover DAQ when one FEE64 stops sending data - short(er) version
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5)   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

6) Start DAQ

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

8) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

9) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

10) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.
  617   Sun Apr 28 15:49:58 2024 TDDAQ restart short(er) cut
16.45 aida04 stopped producing data

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)

       aida04 did not stop, all other FEE64s stopped

      restart Merger (workspace 3, display 2, select 'NewMerger' icon)

      SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
      GO Merger (workspace 3, display 1, browser 'NewMerger' tab)

       aida04 now stops

      disable *all* #1 data transfer 

      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
      check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
      enable *all* #1 data transfer
      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)


      No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
      
  616   Sun Apr 28 15:09:34 2024 Betool Alayed16:0-00:00 28 April 2024

16:00 aida04 not producing data

          power cycle all FEE64s to recover DAQ

16.20 analysis data file R21_668

         max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%

         no timewarps

         HEC data item rate 1.9kHz

 

Attachment 1: R21_668.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256513573 ( 1160512.7 Hz)
        Other data format:    5406427 (   24459.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1079 (       4.9 Hz)
                               RESUME:       1078 (       4.9 Hz)
                              SYNC100:      32824 (     148.5 Hz)
                              WR48-63:      32824 (     148.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5338622 (   24152.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440937 (    1994.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      221.035 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1               19.736                0.000
  2                0.053                0.000
  3               36.607                0.000
  4                0.854                0.000
  5                4.747                0.000
  6                0.000                0.000
  7                1.845                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.661                0.000
 11                0.020                0.000
 12                0.004                0.000
 13                0.000                0.000
 14                0.003                0.000
 15                1.247                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8138410       8624          0          0         12         12        985        985          0       6630          0      38979
  1   27142829     245404          0          0        242        242       3461       3461          0     237998          0      64179
  2   11096772    2350080          0          0          8          8       1768       1768          0    2346528          0      26862
  3   37888832    1835283          0          0        304        303       4762       4762          0    1825152          0      18340
  4    9329455     413600          0          0         28         28       1212       1212          0     411120          0      29227
  5   23846951     517930          0          0        167        167       3201       3201          0     511194          0      24385
  6    5635241       1420          0          0          0          0        710        710          0          0          0       8053
  7   40074524      10318          0          0         93         93       5066       5066          0          0          0     120554
  8    3088839        754          0          0          0          0        377        377          0          0          0      14672
  9    5102920       1254          0          0          0          0        627        627          0          0          0       6080
 10   25963054       6756          0          0        140        140       3238       3238          0          0          0       2640
 11    5980678       1458          0          0          5          5        724        724          0          0          0      52714
 12    5769255       1548          0          0          2          2        772        772          0          0          0       8398
 13    3930590        926          0          0          0          0        463        463          0          0          0       9544
 14    5399163       1328          0          0          1          1        663        663          0          0          0       5692
 15   38126060       9744          0          0         77         77       4795       4795          0          0          0      10618
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.004s (  941.068 blocks/s,  58.817 Mb/s)
  615   Sun Apr 28 06:57:59 2024 Magda Satrazani08:00-16:00 Sunday 28 April

Took over the shift from Marc.

 

 

10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19

          Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD

10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).

         Grafana (AIDA) is now reporting data correctly too (attachment 21).

          Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )

                             Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.

                             Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.

 

14.45 analysis data file R21_500 - attachment 36

         max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%

         no timewarps

         HEC data item rate 2.7kHz

 

Attachment 1: Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Attachment 3: leakage_current.png
leakage_current.png
Attachment 4: Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2024-04-28_at_08-08-14_ucesb.png
Screenshot_2024-04-28_at_08-08-14_ucesb.png
Attachment 6: Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
Attachment 7: Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
Attachment 8: Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Attachment 10: Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
Attachment 11: Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
Attachment 12: Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
Attachment 13: Screenshot_2024-04-28_at_10-04-19_ucesb.png
Screenshot_2024-04-28_at_10-04-19_ucesb.png
Attachment 14: Screenshot_from_2024-04-28_10-08-23.png
Screenshot_from_2024-04-28_10-08-23.png
Attachment 15: Screenshot_from_2024-04-28_10-30-24.png
Screenshot_from_2024-04-28_10-30-24.png
Attachment 16: Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
Attachment 17: Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
Attachment 18: Capture_Grafana.PNG
Capture_Grafana.PNG
Attachment 19: Capture_Grafana_2.PNG
Capture_Grafana_2.PNG
Attachment 20: Screenshot_from_2024-04-28_10-58-43.png
Screenshot_from_2024-04-28_10-58-43.png
Attachment 21: Capture_Grafana_3.PNG
Capture_Grafana_3.PNG
Attachment 22: Screenshot_from_2024-04-28_12-00-49.png
Screenshot_from_2024-04-28_12-00-49.png
Attachment 23: Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
Attachment 24: Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
Attachment 25: Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
Attachment 26: Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
Attachment 27: Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
Attachment 28: Screenshot_2024-04-28_at_12-00-08_ucesb.png
Screenshot_2024-04-28_at_12-00-08_ucesb.png
Attachment 29: Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 30: Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
Attachment 31: Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
Attachment 32: Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
Attachment 33: Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
Attachment 34: Screenshot_2024-04-28_at_14-02-02_ucesb.png
Screenshot_2024-04-28_at_14-02-02_ucesb.png
Attachment 35: Screenshot_from_2024-04-28_14-02-43.png
Screenshot_from_2024-04-28_14-02-43.png
Attachment 36: R21_500.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259329603 ( 2382071.3 Hz)
        Other data format:    2590397 (   23794.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1783 (      16.4 Hz)
                               RESUME:       1783 (      16.4 Hz)
                              SYNC100:      32405 (     297.7 Hz)
                              WR48-63:      32405 (     297.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2522021 (   23166.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     289080 (    2655.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      108.867 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.124                0.000
  1                7.512                0.000
  2                0.941                0.000
  3               17.003                0.000
  4                0.495                0.000
  5                6.767                0.000
  6                0.004                0.000
  7                7.595                0.000
  8                0.000                0.000
  9                0.006                0.000
 10               24.536                0.000
 11                0.145                0.000
 12                0.001                0.000
 13                0.000                0.000
 14                0.322                0.000
 15                8.584                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22169110       8971          0          0         57         57       2822       2822          0       3213          0      16255
  1   12639517     112315          0          0         99         99       1612       1612          0     108893          0      29028
  2   22281064    1127033          0          0         37         37       2985       2985          0    1120989          0      11197
  3   20798124     911187          0          0        135        135       2715       2715          0     905487          0       8456
  4    9363935     176936          0          0         14         14       1205       1205          0     174498          0      11991
  5   27019969     216459          0          0        325        325       3434       3434          0     208941          0       9910
  6    6076361       1558          0          0          2          2        777        777          0          0          0       3470
  7   35565405       8998          0          0        189        189       4310       4310          0          0          0     153479
  8    1477425        362          0          0          0          0        181        181          0          0          0       6096
  9    6250272       1506          0          0          2          2        751        751          0          0          0       2572
 10   27290515       8024          0          0        662        662       3350       3350          0          0          0        840
 11   10865846       2620          0          0         16         16       1294       1294          0          0          0      22001
 12    3883231        948          0          0          2          2        472        472          0          0          0       3485
 13    3455565        826          0          0          0          0        413        413          0          0          0       4071
 14   11216035       2794          0          0         51         51       1346       1346          0          0          0       2348
 15   38977229       9860          0          0        192        192       4738       4738          0          0          0       3881
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.605s (  828.898 blocks/s,  51.806 Mb/s)
  614   Sat Apr 27 23:04:42 2024 Marc00:00-08:00 Sunday 28 April

Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C

0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.

After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.

I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.

I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.

We are now back in business with AIDA back in main DAQ at  01:40

01:45 full checks:

           Bias: AIDA-Graphana - Attachment 1

               Leakage current ramping up again ! S4 temperaturenot really going down either: 25C

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5

      ucesb - attachment 6

03:00 full checks

          No change to be highlighted. All pretty much the same as during previous full checks.

05:00 full checks

Expt still running very smoothly.

           Bias: AIDA-Graphana - Attachment 7-8

               Leakage current going down slightly

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
      per FEE64 Rate spectra - attachment 11

      Merger etc - attachment 12

      ucesb - attachment 13

07:15 all running smoothly.

07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.

07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.

 

Attachment 1: Screenshot_from_2024-04-28_01-51-37.png
Screenshot_from_2024-04-28_01-51-37.png
Attachment 2: Screenshot_from_2024-04-28_01-53-29.png
Screenshot_from_2024-04-28_01-53-29.png
Attachment 3: Screenshot_from_2024-04-28_01-53-52.png
Screenshot_from_2024-04-28_01-53-52.png
Attachment 4: Screenshot_from_2024-04-28_01-54-16.png
Screenshot_from_2024-04-28_01-54-16.png
Attachment 5: Screenshot_from_2024-04-28_01-56-14.png
Screenshot_from_2024-04-28_01-56-14.png
Attachment 6: Screenshot_from_2024-04-28_04-59-46.png
Screenshot_from_2024-04-28_04-59-46.png
Attachment 7: Screenshot_from_2024-04-28_04-59-51.png
Screenshot_from_2024-04-28_04-59-51.png
Attachment 8: Screenshot_from_2024-04-28_05-00-33.png
Screenshot_from_2024-04-28_05-00-33.png
Attachment 9: Screenshot_from_2024-04-28_05-00-25.png
Screenshot_from_2024-04-28_05-00-25.png
Attachment 10: Screenshot_from_2024-04-28_05-00-38.png
Screenshot_from_2024-04-28_05-00-38.png
Attachment 11: Screenshot_from_2024-04-28_05-02-44.png
Screenshot_from_2024-04-28_05-02-44.png
Attachment 12: Screenshot_from_2024-04-28_05-00-03.png
Screenshot_from_2024-04-28_05-00-03.png
  613   Sat Apr 27 15:13:06 2024 Norah16:00 - 00:00 Saturday 27 April

14:50 Checks

Leakage current  is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1

It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.

18:15 Checks

Everything appears to be going smoothly 

Grafana - DSSSD bias  and leakage current - attachments 2 and 3

FEE64 temperatures  - It appears okay, nothing strange , attachment 4

ADC data item stats - attachment 5

Per FEE64 Rate spectra - attachment 6

 ucesb - attachment 7

Merger Link Data Rates  - attachment 8

 

18:16

The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.

18:20  it came back to work normally

............................

20:00 Checks

Nothing new to report. All is well.

20:33 Aida02 showed as not reading - attachment 10

20:34 it resumed reading and returned to normal operation.

21:08 Aida02 was  not reading - attachment 11 , had to emailed Tom to fix it .

The shift crew have restarted ucesb and AIDA  is now showing OK - attachment 12

so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.

ADC data item stats and Merger etc  appear to be going smoothly - attachments 13 and 14

...........................................

22:00 Checks

Still running smoothly.

DSSD bias and  leakage current - ok

FEE64 temperatures - ok

Per FEE64 Rate spectra - ok

ADC data item stats - ok

Ucesb - ok

.................................

23:00 Checks

 Nothing new to report. Everything is going well , and I have captured screenshots of each one

Leakage currents dropped a bit See attachment 15

.............................

23:50 Checks

All seems smooth.

Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
Screenshot_from_2024-04-27_16-54-13.png
Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
Screenshot_from_2024-04-27_18-15-47.png
Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
Screenshot_from_2024-04-27_18-09-13.png
Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
Screenshot_from_2024-04-27_18-07-50.png
Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
Screenshot_from_2024-04-27_18-06-46.png
Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
Screenshot_from_2024-04-27_18-07-19.png
Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
Screenshot_from_2024-04-27_18-08-53.png
Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
Screenshot_from_2024-04-27_18-09-57.png
Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
Screenshot_from_2024-04-27_19-18-03.png
Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
Screenshot_from_2024-04-27_20-33-01.png
Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
Screenshot_from_2024-04-27_20-52-28.png
Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
Screenshot_from_2024-04-27_21-22-51.png
Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
Screenshot_from_2024-04-27_21-44-02.png
Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
Screenshot_from_2024-04-27_21-45-02.png
Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
Screenshot_from_2024-04-27_23-26-06.png
  612   Sat Apr 27 04:59:31 2024 TD08:00-16:00 Saturday 27 April
06.00 DAQ continues file S100_alpha/R20_740
      FRS setting 162Eu

      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage currents tracking S4 ambient temperature

      FEE64 temperatures OK - attachment 3

      All system wide checks OK *except* 
       as reported - attachments 4-5
       *all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L specra - attachments 12-13
       aida09 pulser peak width 55 ch FWHM (~38keV FWHM)

      per FEE64 1.8.H spectra - attachments 14-15

      Merger, TapeServer etc - attachments 16-17
       12Mb/s to disk

      ucesb - attachment 18
       spill length 3.5s, extraction 1.5s

06.30 analysis data file R20_748 - attachment 19
      max deadtime 12%
      no timewarps
      HEC data item rate 2.1kHz

06.35 new run file S100_alpha/R21

09.30 DAQ continues - file S100_alpha/R21_83

      DSSSD bias & leakage currents - Grafana - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats OK - attachments 22

      per FEE64 Rate spectra - attachment 23

      Merger etc - attachments 24

      ucesb - attachment 25
       spill length 3.5s, extraction 1.5s

09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
       no significant change in electronic noise cf. start of run

      per p+n FEE64 1.8.L spectra - attachment 28
       aida09 pulser peak width 62 ch FWHM
       common x/y scale - peak heoght proxy for width


13.57 DAQ continues file R21_141

      DSSSD bias & leakage current - Grafana - attachments 29-30

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachments 32

      per p+n FEE64 1.8.L spectra - attachment 33
       aida09 pulser peak width 55 ch FWHM (~39keV FWHM)

      per FEE64 Rate spectra - attachment 34

      Merger etc - attachments 35

      ucesb - attachment 36
       spill length 3.5s, extraction 1.5s

14.15 analysis data file R21_144 - attachment 35
      max deadtime 16% (aida04)
      no timewarps
      HEC data item rate 2.0kHz
Attachment 1: Screenshot_from_2024-04-27_06-00-39.png
Screenshot_from_2024-04-27_06-00-39.png
Attachment 2: Screenshot_from_2024-04-27_06-01-02.png
Screenshot_from_2024-04-27_06-01-02.png
Attachment 3: Screenshot_from_2024-04-27_06-01-38.png
Screenshot_from_2024-04-27_06-01-38.png
Attachment 4: Screenshot_from_2024-04-27_06-02-22.png
Screenshot_from_2024-04-27_06-02-22.png
Attachment 5: Screenshot_from_2024-04-27_06-02-46.png
Screenshot_from_2024-04-27_06-02-46.png
Attachment 6: Screenshot_from_2024-04-27_06-03-00.png
Screenshot_from_2024-04-27_06-03-00.png
Attachment 7: Screenshot_from_2024-04-27_06-03-32.png
Screenshot_from_2024-04-27_06-03-32.png
Attachment 8: Screenshot_from_2024-04-27_06-04-34.png
Screenshot_from_2024-04-27_06-04-34.png
Attachment 9: Screenshot_from_2024-04-27_06-05-03.png
Screenshot_from_2024-04-27_06-05-03.png
Attachment 10: Screenshot_from_2024-04-27_06-05-37.png
Screenshot_from_2024-04-27_06-05-37.png
Attachment 11: Screenshot_from_2024-04-27_06-06-17.png
Screenshot_from_2024-04-27_06-06-17.png
Attachment 12: Screenshot_from_2024-04-27_06-07-30.png
Screenshot_from_2024-04-27_06-07-30.png
Attachment 13: Screenshot_from_2024-04-27_06-08-12.png
Screenshot_from_2024-04-27_06-08-12.png
Attachment 14: Screenshot_from_2024-04-27_06-11-26.png
Screenshot_from_2024-04-27_06-11-26.png
Attachment 15: Screenshot_from_2024-04-27_06-12-32.png
Screenshot_from_2024-04-27_06-12-32.png
Attachment 16: Screenshot_from_2024-04-27_06-13-07.png
Screenshot_from_2024-04-27_06-13-07.png
Attachment 17: Screenshot_from_2024-04-27_06-13-53.png
Screenshot_from_2024-04-27_06-13-53.png
Attachment 18: Screenshot_from_2024-04-27_06-14-37.png
Screenshot_from_2024-04-27_06-14-37.png
Attachment 19: R20_748
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257504742 ( 1406851.0 Hz)
        Other data format:    4415258 (   24122.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1610 (       8.8 Hz)
                               RESUME:       1611 (       8.8 Hz)
                              SYNC100:      32682 (     178.6 Hz)
                              WR48-63:      32682 (     178.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4346673 (   23747.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     386233 (    2110.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      183.036 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.284                0.000
  1               10.840                0.000
  2                0.188                0.000
  3               23.217                0.000
  4                0.100                0.000
  5                2.070                0.000
  6                0.000                0.000
  7                2.151                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                8.754                0.000
 11                0.009                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.015                0.000
 15                2.919                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12463516       8663          0          0         35         35       1556       1556          0       5481          0      27718
  1   18658776     198614          0          0        164        164       2390       2390          0     193506          0      51705
  2   15562218    1904264          0          0         13         13       2088       2088          0    1900062          0      19441
  3   28663274    1575353          0          0        203        204       3714       3714          0    1567518          0      14524
  4    7649540     297936          0          0          9          9        969        969          0     295980          0      20940
  5   22941092     390586          0          0        220        220       3010       3010          0     384126          0      18537
  6    5869094       1360          0          0          0          0        680        680          0          0          0       5932
  7   40428040      10412          0          0        122        122       5084       5084          0          0          0     148981
  8    2103382        490          0          0          0          0        245        245          0          0          0      10601
  9    5530471       1492          0          0          0          0        746        746          0          0          0       4423
 10   31486867       9242          0          0        711        711       3910       3910          0          0          0       1813
 11    6680681       1722          0          0          2          2        859        859          0          0          0      37346
 12    4842246       1236          0          0          0          0        618        618          0          0          0       5812
 13    3504495        918          0          0          0          0        459        459          0          0          0       6910
 14    6454647       1674          0          0          3          3        834        834          0          0          0       4246
 15   44666403      11296          0          0        128        128       5520       5520          0          0          0       7304
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.639s (  828.186 blocks/s,  51.762 Mb/s)
Attachment 20: Capture.PNG
Capture.PNG
Attachment 21: Screenshot_from_2024-04-27_09-32-29.png
Screenshot_from_2024-04-27_09-32-29.png
Attachment 22: Screenshot_from_2024-04-27_09-33-31.png
Screenshot_from_2024-04-27_09-33-31.png
Attachment 23: Screenshot_from_2024-04-27_09-34-20.png
Screenshot_from_2024-04-27_09-34-20.png
Attachment 24: Screenshot_from_2024-04-27_09-34-58.png
Screenshot_from_2024-04-27_09-34-58.png
Attachment 25: Screenshot_from_2024-04-27_09-35-41.png
Screenshot_from_2024-04-27_09-35-41.png
Attachment 26: Screenshot_from_2024-04-27_09-47-09.png
Screenshot_from_2024-04-27_09-47-09.png
Attachment 27: Screenshot_from_2024-04-27_09-48-06.png
Screenshot_from_2024-04-27_09-48-06.png
Attachment 28: Screenshot_from_2024-04-27_09-52-52.png
Screenshot_from_2024-04-27_09-52-52.png
Attachment 29: Screenshot_from_2024-04-27_13-57-44.png
Screenshot_from_2024-04-27_13-57-44.png
Attachment 30: Screenshot_from_2024-04-27_13-58-01.png
Screenshot_from_2024-04-27_13-58-01.png
Attachment 31: Screenshot_from_2024-04-27_14-01-26.png
Screenshot_from_2024-04-27_14-01-26.png
Attachment 32: Screenshot_from_2024-04-27_13-58-55.png
Screenshot_from_2024-04-27_13-58-55.png
Attachment 33: Screenshot_from_2024-04-27_14-03-49.png
Screenshot_from_2024-04-27_14-03-49.png
Attachment 34: Screenshot_from_2024-04-27_14-04-30.png
Screenshot_from_2024-04-27_14-04-30.png
Attachment 35: Screenshot_from_2024-04-27_14-05-30.png
Screenshot_from_2024-04-27_14-05-30.png
Attachment 36: Screenshot_from_2024-04-27_14-06-42.png
Screenshot_from_2024-04-27_14-06-42.png
Attachment 37: R21_144
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256251858 ( 1109842.9 Hz)
        Other data format:    5668143 (   24549.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1144 (       5.0 Hz)
                               RESUME:       1144 (       5.0 Hz)
                              SYNC100:      32860 (     142.3 Hz)
                              WR48-63:      32860 (     142.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5600135 (   24254.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     452307 (    1959.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      230.890 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.168                0.000
  1               16.872                0.000
  2                0.114                0.000
  3               36.051                0.000
  4                0.888                0.000
  5                2.369                0.000
  6                0.000                0.000
  7                1.443                0.000
  8                0.006                0.000
  9                0.000                0.000
 10                1.304                0.000
 11                0.023                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.033                0.000
 15                1.208                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7949000       9111          0          0         19         19       1076       1076          0       6921          0      39332
  1   26808065     269780          0          0        235        235       3356       3356          0     262598          0      70729
  2   11614466    2448126          0          0         13         13       1817       1817          0    2444466          0      27115
  3   37929923    1952070          0          0        294        294       4896       4896          0    1941690          0      19677
  4    8915163     414541          0          0         12         12       1163       1163          0     412191          0      27553
  5   22729514     538305          0          0        114        114       2904       2904          0     532269          0      25872
  6    5822184       1488          0          0          0          0        744        744          0          0          0       8358
  7   40443466      10506          0          0         99         99       5154       5154          0          0          0     123022
  8    3043811        744          0          0          1          1        371        371          0          0          0      14593
  9    5157550       1312          0          0          0          0        656        656          0          0          0       6148
 10   26044004       6988          0          0        273        273       3221       3221          0          0          0       2629
 11    5888316       1514          0          0          4          4        753        753          0          0          0      52905
 12    5831030       1426          0          0          0          0        713        713          0          0          0       8360
 13    4114996        986          0          0          0          0        493        493          0          0          0       9929
 14    5361222       1380          0          0          5          5        685        685          0          0          0       5844
 15   38599148       9866          0          0         75         75       4858       4858          0          0          0      10241
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.879s (  944.541 blocks/s,  59.034 Mb/s)
  611   Fri Apr 26 23:04:51 2024 Marc00:00-08:00 Saturday 27 April

It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!

0.15 Checks:
      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage current dropped a bit since the evening shift ~22:00.

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

      ucesb - attachment 7

2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12

3:30 Still running smoothly. 

5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.

        DAQ restarted OK. Main DAQ informed.

 

Attachment 1: Screenshot_from_2024-04-27_00-17-00.png
Screenshot_from_2024-04-27_00-17-00.png
Attachment 2: Screenshot_from_2024-04-27_00-16-16.png
Screenshot_from_2024-04-27_00-16-16.png
Attachment 3: Screenshot_from_2024-04-27_00-18-49.png
Screenshot_from_2024-04-27_00-18-49.png
Attachment 4: Screenshot_from_2024-04-27_00-19-05.png
Screenshot_from_2024-04-27_00-19-05.png
Attachment 5: Screenshot_from_2024-04-27_00-19-19.png
Screenshot_from_2024-04-27_00-19-19.png
Attachment 6: Screenshot_from_2024-04-27_00-21-44.png
Screenshot_from_2024-04-27_00-21-44.png
Attachment 7: Screenshot_from_2024-04-27_00-21-00.png
Screenshot_from_2024-04-27_00-21-00.png
Attachment 8: Screenshot_from_2024-04-27_02-32-22.png
Screenshot_from_2024-04-27_02-32-22.png
Attachment 9: Screenshot_from_2024-04-27_02-37-20.png
Screenshot_from_2024-04-27_02-37-20.png
Attachment 10: Screenshot_from_2024-04-27_02-37-31.png
Screenshot_from_2024-04-27_02-37-31.png
Attachment 11: Screenshot_from_2024-04-27_02-40-39.png
Screenshot_from_2024-04-27_02-40-39.png
Attachment 12: Screenshot_from_2024-04-27_02-41-13.png
Screenshot_from_2024-04-27_02-41-13.png
  610   Fri Apr 26 15:34:00 2024 RDP16:00 - 00:00 Friday 26 April

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

 

Attachment 1: Screenshot_2024-04-26_at_16-40-03_ucesb.png
Screenshot_2024-04-26_at_16-40-03_ucesb.png
Attachment 2: Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Attachment 3: Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Attachment 4: Screenshot_2024-04-26_at_19-18-28_ucesb.png
Screenshot_2024-04-26_at_19-18-28_ucesb.png
Attachment 5: Screenshot_2024-04-26_at_22-12-06_ucesb.png
Screenshot_2024-04-26_at_22-12-06_ucesb.png
Attachment 6: Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
  609   Fri Apr 26 08:15:59 2024 TD[How To] Recover DAQ when one FEE64 stops sending data
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.

Typical symptoms are:

- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
   active data links are green, inactive links are red
   N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n

0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5) Reboot problem FEE64s
   Launch a command terminal and telnet to the problem FEE64 for example

    telnet aida04

   Enter user 'root' and enter password. When logged in reboot the FEE64 using the command

    reboot

6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
    you should see two messages showing the FEE64 mounting filesystems
    wait c. three minutes for boot to complete

7) Restart DAQ

   Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
   Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

8) Restore ASIC settings

   Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
   Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
   Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
   Wait 10s

9) Check ASIC settings

   Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
   Wait for popup indicating completion of checks and dismiss

10) System wide checks

    Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']

    Select 'Check clock status'  [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
     make a note or take a screenshot of any error messages observed

11) Start DAQ

    Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
    Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

13) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

14) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

15) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.








    








   

   




   
  608   Fri Apr 26 07:01:26 2024 Norah08:00 - 16:00 Friday 26 April

10:00 Checks:

All seems smooth.

DSSD bias and  leakage current ( Attachment 1)

FEE64 temperatures (Attachment 2)

Per FEE64 Rate spectra (Attachment 3)

ADC data item stats -(Attachment 4)

Ucesb (Attachment 5)

 

10:21

Aida05 was zero not reading had to call Nic to fix it.

10:34

Now, Aida05 is running well.

.......................................................

12:00 Checks:

Everything appears to be going smoothly, and I took screenshot of them.

 

12:28, Aida02 showed as not reading , had to email Nic to fix it.

12:30, it resumed reading and returned to normal operation.

.........................................................

14:00 Checks:

Everything is going well , and I have captured screenshots of each one.

......................................................

16:00 Checks:

All seems smooth, and I took screenshot of them.

 

 

 

 

 

Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
Screenshot_from_2024-04-26_10-00-39.png
Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
Screenshot_from_2024-04-26_10-02-35.png
Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
Screenshot_from_2024-04-26_10-04-40.png
Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
Screenshot_from_2024-04-26_10-06-18.png
Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
Screenshot_from_2024-04-26_10-09-52.png
  607   Thu Apr 25 22:45:30 2024 Tom, Muneerah, NorahOver night shift AIDA 26 Fri

 

01:02

Aida04 was zero not reading had to call Tom to fix it.

02:00 checked all good and attached.

Its been running good after that till 7:30 again the same problem happened and Tom fixed it.

 

 

Attachment 1: Screenshot_from_2024-04-25_23-42-15.png
Screenshot_from_2024-04-25_23-42-15.png
Attachment 2: Screenshot_from_2024-04-26_00-05-47.png
Screenshot_from_2024-04-26_00-05-47.png
Attachment 3: Screenshot_from_2024-04-26_00-05-15.png
Screenshot_from_2024-04-26_00-05-15.png
Attachment 4: Screenshot_from_2024-04-26_00-04-22.png
Screenshot_from_2024-04-26_00-04-22.png
Attachment 5: Screenshot_from_2024-04-26_02-00-17.png
Screenshot_from_2024-04-26_02-00-17.png
Attachment 6: Screenshot_from_2024-04-26_01-58-39.png
Screenshot_from_2024-04-26_01-58-39.png
Attachment 7: Screenshot_from_2024-04-26_01-58-39.png
Screenshot_from_2024-04-26_01-58-39.png
Attachment 8: Screenshot_from_2024-04-26_01-57-38.png
Screenshot_from_2024-04-26_01-57-38.png
Attachment 9: Screenshot_from_2024-04-26_01-57-01.png
Screenshot_from_2024-04-26_01-57-01.png
  606   Thu Apr 25 15:02:50 2024 Marc16:00 - 0:00 Thursday 25 April
16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

 

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

Attachment 1: Screenshot_from_2024-04-25_16-19-32.png
Screenshot_from_2024-04-25_16-19-32.png
Attachment 2: Screenshot_from_2024-04-25_16-45-19.png
Screenshot_from_2024-04-25_16-45-19.png
Attachment 3: Screenshot_from_2024-04-25_16-46-12.png
Screenshot_from_2024-04-25_16-46-12.png
Attachment 4: Screenshot_from_2024-04-25_16-46-41.png
Screenshot_from_2024-04-25_16-46-41.png
Attachment 5: Screenshot_from_2024-04-25_16-47-00.png
Screenshot_from_2024-04-25_16-47-00.png
Attachment 6: Screenshot_from_2024-04-25_16-52-18.png
Screenshot_from_2024-04-25_16-52-18.png
Attachment 7: Screenshot_from_2024-04-25_16-52-50.png
Screenshot_from_2024-04-25_16-52-50.png
Attachment 8: Screenshot_from_2024-04-25_17-23-36.png
Screenshot_from_2024-04-25_17-23-36.png
  605   Thu Apr 25 08:26:50 2024 TD08:00-16:00 Thursday 25 April
09.56 DAQ continues data file S100_alpha/R18_387
      FRS switching settings

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK  - attachment 2
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 3-5

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10

      per FEE64 Rate spectra - attachments 11

      per p+n FEE64 1.8.L spectra - attachments 12-13
       aida09 pulser peak width 57 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 14

      per FEE64 1.8.H spectra - attachments 15-16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 19-21

      BNC PB-5 settings - attachment 22

10.07 data file S100_alpha/R20
      new run for new FRS setting

10.08 all histograms, stats, merger stats and data link stats zero'd

10.10 ADC data item stats (beam OFF) - attachment 23
      11/16 < 20k, max 129k (aida08)

10.31 analysis of data file R18_349 (closed 06.00 this morning)
      max deadtime 12% (aida04)
      no timewarps
      HEC data item rate 1.7kHz

11.00 tuning 168dy FRS setting
      ucesb - attachment 24
      spill length 3.5s, extraction time 1.5s

11.30 degraders now set to implant in AIDA DSSSDs
      high rates expected 

      per FEE64 Rate spectra - attachment 25

      ADC data item stats - attachment 26

      Merger etc - attachment 27
      in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged

      DSSSD bias & leakage current - attachment 28
       c. few hundred nA step in leakage current 

      analysis of data file R20_33 - attachment 29
      max deadtime 18% (aida08)
      no timewarps
      HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)

      Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
      SC42 L/R currently report c. 4.5kHz

      Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
      If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill

      we are probably somewhere in between

12.50 DSSSD bias & leakage current - Grafana - attachment 30
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 31

      ADC data item stats - attachment 32

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

14.56 analysis of data file R20_106 - attachment 35
      max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
      no timewarps
      HEC data item rate 4.2kHz


15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41

      ucesb - attachment 42

15.57 all histograms zero'd




      
Attachment 1: Screenshot_from_2024-04-25_09-27-20.png
Screenshot_from_2024-04-25_09-27-20.png
Attachment 2: Screenshot_from_2024-04-25_09-27-44.png
Screenshot_from_2024-04-25_09-27-44.png
Attachment 3: Screenshot_from_2024-04-25_09-28-05.png
Screenshot_from_2024-04-25_09-28-05.png
Attachment 4: Screenshot_from_2024-04-25_09-28-16.png
Screenshot_from_2024-04-25_09-28-16.png
Attachment 5: Screenshot_from_2024-04-25_09-28-23.png
Screenshot_from_2024-04-25_09-28-23.png
Attachment 6: Screenshot_from_2024-04-25_09-29-04.png
Screenshot_from_2024-04-25_09-29-04.png
Attachment 7: Screenshot_from_2024-04-25_09-29-20.png
Screenshot_from_2024-04-25_09-29-20.png
Attachment 8: Screenshot_from_2024-04-25_09-29-46.png
Screenshot_from_2024-04-25_09-29-46.png
Attachment 9: Screenshot_from_2024-04-25_09-29-59.png
Screenshot_from_2024-04-25_09-29-59.png
Attachment 10: Screenshot_from_2024-04-25_09-30-15.png
Screenshot_from_2024-04-25_09-30-15.png
Attachment 11: Screenshot_from_2024-04-25_09-30-56.png
Screenshot_from_2024-04-25_09-30-56.png
Attachment 12: Screenshot_from_2024-04-25_09-31-55.png
Screenshot_from_2024-04-25_09-31-55.png
Attachment 13: Screenshot_from_2024-04-25_09-32-11.png
Screenshot_from_2024-04-25_09-32-11.png
Attachment 14: Screenshot_from_2024-04-25_09-33-15.png
Screenshot_from_2024-04-25_09-33-15.png
Attachment 15: Screenshot_from_2024-04-25_09-34-23.png
Screenshot_from_2024-04-25_09-34-23.png
Attachment 16: Screenshot_from_2024-04-25_09-34-52.png
Screenshot_from_2024-04-25_09-34-52.png
Attachment 17: Screenshot_from_2024-04-25_09-35-38.png
Screenshot_from_2024-04-25_09-35-38.png
Attachment 18: Screenshot_from_2024-04-25_09-36-14.png
Screenshot_from_2024-04-25_09-36-14.png
Attachment 19: Screenshot_from_2024-04-25_09-36-28.png
Screenshot_from_2024-04-25_09-36-28.png
Attachment 20: Screenshot_from_2024-04-25_09-37-02.png
Screenshot_from_2024-04-25_09-37-02.png
Attachment 21: Screenshot_from_2024-04-25_09-37-17.png
Screenshot_from_2024-04-25_09-37-17.png
Attachment 22: Screenshot_from_2024-04-25_10-02-31.png
Screenshot_from_2024-04-25_10-02-31.png
Attachment 23: R18_349
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254055879 (  798496.4 Hz)
        Other data format:    7864121 (   24716.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        850 (       2.7 Hz)
                               RESUME:        851 (       2.7 Hz)
                              SYNC100:      33185 (     104.3 Hz)
                              WR48-63:      33185 (     104.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7796050 (   24502.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     547807 (    1721.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.168 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                0.000
  1               15.547                0.000
  2                0.026                0.000
  3               39.441                0.000
  4                0.410                0.000
  5                0.954                0.000
  6                0.000                0.000
  7                0.601                0.000
  8                0.003                0.000
  9                0.000                0.000
 10                0.050                0.000
 11                0.014                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.116                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6649451      11230          0          0          3          3        839        839          0       9546          0      49433
  1   33550123     378474          0          0        245        245       4373       4373          0     369238          0     100748
  2    7942952    3356106          0          0          5          5       1429       1429          0    3353238          0      35304
  3   50066846    2782463          0          0        379        380       6547       6547          0    2768610          0      28219
  4   11634489     554162          0          0         19         19       1542       1542          0     551040          0      47658
  5   18478696     749338          0          0         77         77       2403       2403          0     744378          0      37331
  6    6034652       1500          0          0          0          0        750        750          0          0          0      11345
  7   41275133      10854          0          0         71         71       5356       5356          0          0          0      92779
  8    3808950        934          0          0          2          2        465        465          0          0          0      19548
  9    5816078       1448          0          0          0          0        724        724          0          0          0       8443
 10   16595017       4342          0          0         16         16       2155       2155          0          0          0       3675
 11    5732285       1360          0          0          2          2        678        678          0          0          0      65221
 12    7170732       1770          0          0          0          0        885        885          0          0          0      12515
 13    5117877       1350          0          0          0          0        675        675          0          0          0      13502
 14    5096608       1234          0          0          2          2        615        615          0          0          0       7339
 15   29085990       7556          0          0         29         29       3749       3749          0          0          0      14747
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.840s ( 1005.030 blocks/s,  62.814 Mb/s)
Attachment 24: Screenshot_from_2024-04-25_11-27-43.png
Screenshot_from_2024-04-25_11-27-43.png
Attachment 25: Screenshot_from_2024-04-25_11-39-59.png
Screenshot_from_2024-04-25_11-39-59.png
Attachment 26: Screenshot_from_2024-04-25_11-40-11.png
Screenshot_from_2024-04-25_11-40-11.png
Attachment 27: Screenshot_from_2024-04-25_11-40-28.png
Screenshot_from_2024-04-25_11-40-28.png
Attachment 28: Screenshot_from_2024-04-25_11-43-22.png
Screenshot_from_2024-04-25_11-43-22.png
Attachment 29: R20_33
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257467881 ( 1603138.0 Hz)
        Other data format:    4452119 (   27721.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1792 (      11.2 Hz)
                               RESUME:       1788 (      11.1 Hz)
                              SYNC100:      32598 (     203.0 Hz)
                              WR48-63:      32598 (     203.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4383343 (   27293.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659812 (    4108.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.602 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                4.032                0.000
  1               14.271                0.000
  2               17.643                0.000
  3               26.583                0.000
  4               10.476                0.000
  5               15.929                0.000
  6                1.815                0.000
  7               29.387                0.000
  8                0.158                0.000
  9                0.036                0.000
 10                0.592                0.000
 11                0.531                0.000
 12               14.310                0.000
 13                0.165                0.000
 14                0.189                0.000
 15                0.995                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14126134       8441          0          0         70         70       1791       1791          0       4719          0      78615
  1   16495785     288035          0          0        163        162       2094       2094          0     283522          0     105734
  2   19988207    1651337          0          0        161        161       2732       2732          0    1645551          0      49977
  3   24594867    1476087          0          0        215        214       3183       3183          0    1469292          0      26591
  4   15580694     468076          0          0        119        119       1941       1941          0     463956          0      27262
  5   22750595     522722          0          0        300        299       2910       2910          0     516303          0      62407
  6   11629769       3045          0          0         37         36       1486       1486          0          0          0      11991
  7   37179788       9710          0          0        273        273       4582       4582          0          0          0      86342
  8    7628214       1994          0          0          9          9        988        988          0          0          0       1179
  9    5725517       1418          0          0          5          5        704        704          0          0          0       1622
 10   13190745       3550          0          0         69         69       1706       1706          0          0          0         70
 11    9285908       2326          0          0         29         29       1134       1134          0          0          0      74999
 12   16713447       4700          0          0        252        252       2098       2098          0          0          0      21646
 13    6115493       1498          0          0         25         25        724        724          0          0          0      35370
 14    8494444       2072          0          0         11         11       1025       1025          0          0          0         92
 15   27968274       7108          0          0         54         54       3500       3500          0          0          0      75915
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.629s (  924.083 blocks/s,  57.755 Mb/s)
Attachment 30: Screenshot_from_2024-04-25_12-50-09.png
Screenshot_from_2024-04-25_12-50-09.png
Attachment 31: Screenshot_from_2024-04-25_12-50-16.png
Screenshot_from_2024-04-25_12-50-16.png
Attachment 32: Screenshot_from_2024-04-25_12-50-28.png
Screenshot_from_2024-04-25_12-50-28.png
Attachment 33: Screenshot_from_2024-04-25_12-50-35.png
Screenshot_from_2024-04-25_12-50-35.png
Attachment 34: Screenshot_from_2024-04-25_12-51-02.png
Screenshot_from_2024-04-25_12-51-02.png
Attachment 35: R20_106
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257522900 ( 1655989.1 Hz)
        Other data format:    4397100 (   28275.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1842 (      11.8 Hz)
                               RESUME:       1844 (      11.9 Hz)
                              SYNC100:      32579 (     209.5 Hz)
                              WR48-63:      32579 (     209.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4328256 (   27832.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659185 (    4238.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      155.510 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                3.085                0.000
  1               13.568                0.000
  2               18.124                0.000
  3               27.697                0.000
  4                9.792                0.000
  5               16.474                0.000
  6                2.671                0.000
  7               28.543                0.000
  8                0.065                0.000
  9                0.015                0.000
 10                0.643                0.000
 11                0.597                0.000
 12               14.389                0.000
 13                0.083                0.000
 14                0.439                0.000
 15                1.481                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13689199       8238          0          0         65         65       1771       1771          0       4566          0      77953
  1   16947527     287988          0          0        165        166       2170       2170          0     283317          0     104868
  2   20356284    1630731          0          0        174        175       2797       2797          0    1624788          0      51128
  3   23896908    1435934          0          0        213        214       3077       3077          0    1429353          0      26122
  4   15647459     464207          0          0        117        117       2008       2008          0     459957          0      27024
  5   22696175     532714          0          0        306        305       2914       2914          0     526275          0      64287
  6   12236208       3082          0          0         42         42       1499       1499          0          0          0      11994
  7   36272075       9624          0          0        259        259       4553       4553          0          0          0      84854
  8    7567267       1906          0          0          4          4        949        949          0          0          0       1286
  9    5993294       1442          0          0          3          3        718        718          0          0          0       1730
 10   13223300       3544          0          0         82         82       1690       1690          0          0          0         53
 11    9630981       2436          0          0         36         36       1182       1182          0          0          0      75223
 12   16792007       4646          0          0        270        270       2053       2053          0          0          0      21202
 13    6155506       1514          0          0         19         19        738        738          0          0          0      35333
 14    8544527       2006          0          0         19         19        984        984          0          0          0         68
 15   27874183       7088          0          0         68         68       3476       3476          0          0          0      76060
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.426s ( 1051.740 blocks/s,  65.734 Mb/s)
Attachment 36: Screenshot_from_2024-04-25_15-39-07.png
Screenshot_from_2024-04-25_15-39-07.png
Attachment 37: Screenshot_from_2024-04-25_15-39-14.png
Screenshot_from_2024-04-25_15-39-14.png
Attachment 38: Screenshot_from_2024-04-25_15-39-36.png
Screenshot_from_2024-04-25_15-39-36.png
Attachment 39: Screenshot_from_2024-04-25_15-39-53.png
Screenshot_from_2024-04-25_15-39-53.png
Attachment 40: Screenshot_from_2024-04-25_15-40-30.png
Screenshot_from_2024-04-25_15-40-30.png
Attachment 41: Screenshot_from_2024-04-25_15-40-48.png
Screenshot_from_2024-04-25_15-40-48.png
Attachment 42: Screenshot_from_2024-04-25_15-43-08.png
Screenshot_from_2024-04-25_15-43-08.png
  604   Wed Apr 24 21:39:51 2024 TD00:00-08:00 Thursday 25 April
22.43 24.4.24

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

      analysis of data file R18_266 - attachment 6
       max dead time 12% (aida04)
       no timewarps
       HEC data item rate 1.7kHz

03.37 DSSSD bias & leakage current OK - attachments 7-8

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10

      per FEE64 rate spectra - attachments 11

      Merger etc - attachment 12
Attachment 1: Screenshot_from_2024-04-24_22-43-23.png
Screenshot_from_2024-04-24_22-43-23.png
Attachment 2: Screenshot_from_2024-04-24_22-43-58.png
Screenshot_from_2024-04-24_22-43-58.png
Attachment 3: Screenshot_from_2024-04-24_22-44-50.png
Screenshot_from_2024-04-24_22-44-50.png
Attachment 4: Screenshot_from_2024-04-24_22-45-35.png
Screenshot_from_2024-04-24_22-45-35.png
Attachment 5: Screenshot_from_2024-04-24_22-46-21.png
Screenshot_from_2024-04-24_22-46-21.png
Attachment 6: R18_266
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254416606 (  820824.6 Hz)
        Other data format:    7503396 (   24208.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        846 (       2.7 Hz)
                               RESUME:        846 (       2.7 Hz)
                              SYNC100:      33155 (     107.0 Hz)
                              WR48-63:      33155 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7435394 (   23988.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     524404 (    1691.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.014                0.000
  1               16.009                0.000
  2                0.003                0.000
  3               37.388                0.000
  4                0.176                0.000
  5                0.544                0.000
  6                0.000                0.000
  7                0.784                0.000
  8                0.015                0.000
  9                0.000                0.000
 10                0.083                0.000
 11                0.004                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.227                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6529814      10932          0          0          4          4        812        812          0       9300          0      46302
  1   32238460     350302          0          0        231        231       4129       4129          0     341582          0      92776
  2    8209450    3208628          0          0          2          2       1421       1421          0    3205782          0      32990
  3   48353626    2676061          0          0        374        374       6369       6369          0    2662575          0      27015
  4   10911439     520528          0          0          9          9       1454       1454          0     517602          0      45169
  5   18550590     703657          0          0         64         64       2488       2488          0     698553          0      35506
  6    6051492       1578          0          0          0          0        789        789          0          0          0      10690
  7   41812278      11146          0          0         85         85       5488       5488          0          0          0      97961
  8    3594476        900          0          0          4          4        446        446          0          0          0      18281
  9    5838926       1490          0          0          0          0        745        745          0          0          0       8125
 10   18621964       4722          0          0         27         27       2334       2334          0          0          0       3637
 11    5447469       1386          0          0          1          1        692        692          0          0          0      60590
 12    7038926       1642          0          0          0          0        821        821          0          0          0      12101
 13    5011795       1240          0          0          0          0        620        620          0          0          0      12487
 14    5028304       1286          0          0          2          2        641        641          0          0          0       7019
 15   31177597       7898          0          0         43         43       3906       3906          0          0          0      13755
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.508s (  984.379 blocks/s,  61.524 Mb/s)
Attachment 7: Screenshot_from_2024-04-25_03-37-40.png
Screenshot_from_2024-04-25_03-37-40.png
Attachment 8: Screenshot_from_2024-04-25_03-38-57.png
Screenshot_from_2024-04-25_03-38-57.png
Attachment 9: Screenshot_from_2024-04-25_03-39-38.png
Screenshot_from_2024-04-25_03-39-38.png
Attachment 10: Screenshot_from_2024-04-25_03-40-20.png
Screenshot_from_2024-04-25_03-40-20.png
Attachment 11: Screenshot_from_2024-04-25_03-40-52.png
Screenshot_from_2024-04-25_03-40-52.png
Attachment 12: Screenshot_from_2024-04-25_03-41-31.png
Screenshot_from_2024-04-25_03-41-31.png
  603   Wed Apr 24 18:44:05 2024 PP19:30 checks

All seems smooth.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_19-34-58.png
Screenshot_from_2024-04-24_19-34-58.png
Attachment 2: Screenshot_from_2024-04-24_19-37-12.png
Screenshot_from_2024-04-24_19-37-12.png
Attachment 3: Screenshot_from_2024-04-24_19-40-36.png
Screenshot_from_2024-04-24_19-40-36.png
Attachment 4: Screenshot_from_2024-04-24_19-41-39.png
Screenshot_from_2024-04-24_19-41-39.png
Attachment 5: Screenshot_from_2024-04-24_19-42-20.png
Screenshot_from_2024-04-24_19-42-20.png
Attachment 6: Screenshot_from_2024-04-24_19-42-53.png
Screenshot_from_2024-04-24_19-42-53.png
Attachment 7: Screenshot_from_2024-04-24_19-43-21.png
Screenshot_from_2024-04-24_19-43-21.png
Attachment 8: Screenshot_from_2024-04-24_19-43-41.png
Screenshot_from_2024-04-24_19-43-41.png
  602   Wed Apr 24 15:37:59 2024 PP16:00-00:00 shift Wednesday 24 April

All seems OK.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_16-38-45.png
Screenshot_from_2024-04-24_16-38-45.png
Attachment 2: Screenshot_from_2024-04-24_16-39-09.png
Screenshot_from_2024-04-24_16-39-09.png
Attachment 3: Screenshot_from_2024-04-24_16-39-38.png
Screenshot_from_2024-04-24_16-39-38.png
Attachment 4: Screenshot_from_2024-04-24_16-40-24.png
Screenshot_from_2024-04-24_16-40-24.png
Attachment 5: Screenshot_from_2024-04-24_16-41-15.png
Screenshot_from_2024-04-24_16-41-15.png
Attachment 6: Screenshot_from_2024-04-24_16-43-41.png
Screenshot_from_2024-04-24_16-43-41.png
Attachment 7: Screenshot_from_2024-04-24_16-44-15.png
Screenshot_from_2024-04-24_16-44-15.png
Attachment 8: Screenshot_from_2024-04-24_16-44-32.png
Screenshot_from_2024-04-24_16-44-32.png
  601   Wed Apr 24 09:04:55 2024 TD08:00-16:00 Wednesday 24 April
09.56 DAQ continues data file S100_alpha/R18_122
      FRS setting 162Eu

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK  - attachment 3
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 4-6

      WR timestamps OK - attachment 7

      ASIC settings aida01 & aida02 - attachments 8-9

      ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15

      per FEE64 Rate spectra - attachments 16-18
       unusually can observe HEC Rates esp. in DSSSD #1 

      per p+n FEE64 1.8.L spectra - attachments 19-21
       aida09 pulser peak width 61 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 22

      per FEE64 1.8.H spectra - attachments 23-24

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 27-28

      ucesb - attachments 29-30
       spill cycle reported as 3.5s, extraction 1.5s

10.36 All histograms & stats zero'd

      BNC PB-5 settings - attachment 31

      DSSSD bias & leakage current - Grafana Alerting - attachment 32
       can observe bias & leakage current in greater detail
       DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
       DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1

10.46 analysis of data file R18_131
      max dead time 12% aida04, 6% aida02, all others 0.1% or less
      no timewarps
      HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
      SC42 L/R currently report c. 2kHz





   

      
Attachment 1: Screenshot_from_2024-04-24_09-56-46.png
Screenshot_from_2024-04-24_09-56-46.png
Attachment 2: Screenshot_from_2024-04-24_09-57-04.png
Screenshot_from_2024-04-24_09-57-04.png
Attachment 3: Screenshot_from_2024-04-24_09-57-42.png
Screenshot_from_2024-04-24_09-57-42.png
Attachment 4: Screenshot_from_2024-04-24_09-57-57.png
Screenshot_from_2024-04-24_09-57-57.png
Attachment 5: Screenshot_from_2024-04-24_09-58-05.png
Screenshot_from_2024-04-24_09-58-05.png
Attachment 6: Screenshot_from_2024-04-24_09-58-19.png
Screenshot_from_2024-04-24_09-58-19.png
Attachment 7: Screenshot_from_2024-04-24_09-58-57.png
Screenshot_from_2024-04-24_09-58-57.png
Attachment 8: Screenshot_from_2024-04-24_09-59-09.png
Screenshot_from_2024-04-24_09-59-09.png
Attachment 9: Screenshot_from_2024-04-24_09-59-20.png
Screenshot_from_2024-04-24_09-59-20.png
Attachment 10: Screenshot_from_2024-04-24_09-59-47.png
Screenshot_from_2024-04-24_09-59-47.png
Attachment 11: Screenshot_from_2024-04-24_09-59-59.png
Screenshot_from_2024-04-24_09-59-59.png
Attachment 12: Screenshot_from_2024-04-24_10-00-21.png
Screenshot_from_2024-04-24_10-00-21.png
Attachment 13: Screenshot_from_2024-04-24_10-00-37.png
Screenshot_from_2024-04-24_10-00-37.png
Attachment 14: Screenshot_from_2024-04-24_10-00-51.png
Screenshot_from_2024-04-24_10-00-51.png
Attachment 15: Screenshot_from_2024-04-24_10-01-06.png
Screenshot_from_2024-04-24_10-01-06.png
Attachment 16: Screenshot_from_2024-04-24_10-01-31.png
Screenshot_from_2024-04-24_10-01-31.png
Attachment 17: Screenshot_from_2024-04-24_10-01-39.png
Screenshot_from_2024-04-24_10-01-39.png
Attachment 18: Screenshot_from_2024-04-24_10-02-48.png
Screenshot_from_2024-04-24_10-02-48.png
Attachment 19: Screenshot_from_2024-04-24_10-03-43.png
Screenshot_from_2024-04-24_10-03-43.png
Attachment 20: Screenshot_from_2024-04-24_10-04-03.png
Screenshot_from_2024-04-24_10-04-03.png
Attachment 21: Screenshot_from_2024-04-24_10-04-24.png
Screenshot_from_2024-04-24_10-04-24.png
Attachment 22: Screenshot_from_2024-04-24_10-06-03.png
Screenshot_from_2024-04-24_10-06-03.png
Attachment 23: Screenshot_from_2024-04-24_10-07-09.png
Screenshot_from_2024-04-24_10-07-09.png
Attachment 24: Screenshot_from_2024-04-24_10-08-01.png
Screenshot_from_2024-04-24_10-08-01.png
Attachment 25: Screenshot_from_2024-04-24_10-08-56.png
Screenshot_from_2024-04-24_10-08-56.png
Attachment 26: Screenshot_from_2024-04-24_10-09-21.png
Screenshot_from_2024-04-24_10-09-21.png
Attachment 27: Screenshot_from_2024-04-24_10-10-39.png
Screenshot_from_2024-04-24_10-10-39.png
Attachment 28: Screenshot_from_2024-04-24_10-10-47.png
Screenshot_from_2024-04-24_10-10-47.png
Attachment 29: Screenshot_from_2024-04-24_10-11-34.png
Screenshot_from_2024-04-24_10-11-34.png
Attachment 30: Screenshot_from_2024-04-24_10-11-40.png
Screenshot_from_2024-04-24_10-11-40.png
Attachment 31: Screenshot_from_2024-04-24_10-36-43.png
Screenshot_from_2024-04-24_10-36-43.png
Attachment 32: Screenshot_from_2024-04-24_10-38-18.png
Screenshot_from_2024-04-24_10-38-18.png
Attachment 33: R18_131
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254316594 (  804922.0 Hz)
        Other data format:    7603408 (   24065.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        827 (       2.6 Hz)
                               RESUME:        827 (       2.6 Hz)
                              SYNC100:      33177 (     105.0 Hz)
                              WR48-63:      33177 (     105.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7535400 (   23849.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     512028 (    1620.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      315.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               17.612                0.000
  2                0.021                0.000
  3               36.829                0.000
  4                0.071                0.000
  5                0.375                0.000
  6                0.000                0.000
  7                0.653                0.000
  8                0.004                0.000
  9                0.000                0.000
 10                0.011                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.010                0.000
 15                0.281                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5990261      11033          0          0          0          0        778        778          0       9477          0      48652
  1   32634474     342633          0          0        250        249       4180       4180          0     333774          0      88686
  2    8233792    3266215          0          0          3          3       1445       1445          0    3263319          0      33897
  3   49889631    2758758          0          0        373        374       6684       6684          0    2744643          0      26463
  4    7710704     511890          0          0          7          7       1031       1031          0     509814          0      33167
  5   17424106     678891          0          0         57         57       2202       2202          0     674373          0      32013
  6    6276423       1654          0          0          0          0        827        827          0          0          0       9743
  7   42348474      10910          0          0         79         79       5376       5376          0          0          0     105118
  8    3355535        852          0          0          1          1        425        425          0          0          0      18387
  9    5876369       1434          0          0          0          0        717        717          0          0          0       7335
 10   19688353       5108          0          0          5          5       2549       2549          0          0          0       3256
 11    5133774       1300          0          0          1          1        649        649          0          0          0      64128
 12    6809941       1758          0          0          0          0        879        879          0          0          0       9510
 13    5093601       1298          0          0          0          0        649        649          0          0          0      11354
 14    4801692       1232          0          0          1          1        615        615          0          0          0       7416
 15   33049464       8442          0          0         50         50       4171       4171          0          0          0      12903
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.441s ( 1086.905 blocks/s,  67.932 Mb/s)
  600   Tue Apr 23 22:37:46 2024 TD00:00-08:00 Wednesday 24 April
23.34 23.4.24

      DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

03.41 fDSSSD bias & leakage current OK - attachment 6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8

      per FEE64 rate spectra (off and on spill) - attachments 9-10

      Merger etc - attachment 11

03.54 analysis data file R18_67 - attachment 12
      max deadtime 11% (aida04)
      HEC data item rate c. 1.5kHz
      no timewarps

0.35 fDSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data item stats - attachment 15

      per FEE64 rate spectra - attachments 16

      Merger etc - attachment 17

03.54 analysis data file R18_96 - attachment 18
      max deadtime 12% (aida04)
      HEC data item rate c. 1.6kHz
      no timewarps
Attachment 1: Screenshot_from_2024-04-23_23-34-18.png
Screenshot_from_2024-04-23_23-34-18.png
Attachment 2: Screenshot_from_2024-04-23_23-34-54.png
Screenshot_from_2024-04-23_23-34-54.png
Attachment 3: Screenshot_from_2024-04-23_23-35-45.png
Screenshot_from_2024-04-23_23-35-45.png
Attachment 4: Screenshot_from_2024-04-23_23-36-31.png
Screenshot_from_2024-04-23_23-36-31.png
Attachment 5: Screenshot_from_2024-04-23_23-37-21.png
Screenshot_from_2024-04-23_23-37-21.png
Attachment 6: Screenshot_from_2024-04-24_03-41-40.png
Screenshot_from_2024-04-24_03-41-40.png
Attachment 7: Screenshot_from_2024-04-24_03-42-32.png
Screenshot_from_2024-04-24_03-42-32.png
Attachment 8: Screenshot_from_2024-04-24_03-42-58.png
Screenshot_from_2024-04-24_03-42-58.png
Attachment 9: Screenshot_from_2024-04-24_03-43-35.png
Screenshot_from_2024-04-24_03-43-35.png
Attachment 10: Screenshot_from_2024-04-24_03-44-11.png
Screenshot_from_2024-04-24_03-44-11.png
Attachment 11: Screenshot_from_2024-04-24_03-44-35.png
Screenshot_from_2024-04-24_03-44-35.png
Attachment 12: R18_67
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250287937 (  518744.3 Hz)
        Other data format:   11632065 (   24108.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1106 (       2.3 Hz)
                               RESUME:       1109 (       2.3 Hz)
                              SYNC100:      33796 (      70.0 Hz)
                              WR48-63:      33796 (      70.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11562258 (   23963.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     752941 (    1560.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      482.488 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               23.222                0.000
  2                0.000                0.000
  3               56.625                0.000
  4                0.000                0.000
  5                0.704                0.000
  6                0.000                0.000
  7                0.939                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.006                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3375676      15359          0          0          0          0        442        442          0      14475          0      75815
  1   47916692     537101          0          0        366        367       6254       6254          0     523860          0     139250
  2    3817331    4998323          0          0          0          0       1132       1132          0    4996059          0      51826
  3   75749746    4208495          0          0        548        550      10285      10285          0    4186827          0      40621
  4    3336977     792495          0          0          1          1        545        545          0     791403          0      53742
  5   23132211    1055976          0          0         83         83       3088       3088          0    1049634          0      49833
  6    3422365        890          0          0          0          0        445        445          0          0          0      14690
  7   56668346      14890          0          0        106        106       7339       7339          0          0          0     118883
  8    2824304        672          0          0          0          0        336        336          0          0          0      28810
  9    3844587        966          0          0          0          0        483        483          0          0          0      11512
 10    4370286       1248          0          0          0          0        624        624          0          0          0       5026
 11    3088851        802          0          0          0          0        401        401          0          0          0      98441
 12    5014612       1352          0          0          0          0        676        676          0          0          0      15213
 13    3462527        896          0          0          0          0        448        448          0          0          0      18001
 14    3185286        772          0          0          2          2        384        384          0          0          0      11128
 15    7078140       1828          0          0          0          0        914        914          0          0          0      20150
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.998s ( 1032.323 blocks/s,  64.520 Mb/s)
Attachment 13: Screenshot_from_2024-04-24_07-35-18.png
Screenshot_from_2024-04-24_07-35-18.png
Attachment 14: Screenshot_from_2024-04-24_07-36-00.png
Screenshot_from_2024-04-24_07-36-00.png
Attachment 15: Screenshot_from_2024-04-24_07-36-29.png
Screenshot_from_2024-04-24_07-36-29.png
Attachment 16: Screenshot_from_2024-04-24_07-37-03.png
Screenshot_from_2024-04-24_07-37-03.png
Attachment 17: Screenshot_from_2024-04-24_07-37-45.png
Screenshot_from_2024-04-24_07-37-45.png
Attachment 18: R18_96
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253809078 (  751826.0 Hz)
        Other data format:    8110924 (   24025.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        844 (       2.5 Hz)
                               RESUME:        844 (       2.5 Hz)
                              SYNC100:      33258 (      98.5 Hz)
                              WR48-63:      33258 (      98.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8042720 (   23823.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     529514 (    1568.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      337.590 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               19.161                0.000
  2                0.007                0.000
  3               41.205                0.000
  4                0.172                0.000
  5                0.406                0.000
  6                0.000                0.000
  7                0.656                0.000
  8                0.010                0.000
  9                0.000                0.000
 10                0.193                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.023                0.000
 15                0.121                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5871650      11620          0          0          0          0        746        746          0      10128          0      52136
  1   34432367     363819          0          0        263        263       4482       4482          0     354329          0      94194
  2    7680769    3497668          0          0          2          2       1401       1401          0    3494862          0      35954
  3   53024609    2927317          0          0        398        398       7008       7008          0    2912505          0      27672
  4    8502384     548708          0          0          9          9       1159       1159          0     546372          0      36407
  5   16750131     728930          0          0         53         53       2150       2150          0     724524          0      33989
  6    6256772       1674          0          0          0          0        837        837          0          0          0      10191
  7   41893078      10740          0          0         71         71       5299       5299          0          0          0      94711
  8    3653651        920          0          0          2          2        458        458          0          0          0      20092
  9    5991221       1564          0          0          0          0        782        782          0          0          0       8149
 10   16753048       4406          0          0          5          5       2198       2198          0          0          0       3402
 11    5222175       1254          0          0          1          1        626        626          0          0          0      68583
 12    7072961       1726          0          0          0          0        863        863          0          0          0      10498
 13    5337727       1396          0          0          0          0        698        698          0          0          0      12368
 14    4940099       1278          0          0          4          4        635        635          0          0          0       7688
 15   30426436       7904          0          0         36         36       3916       3916          0          0          0      13480
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.307s ( 1055.874 blocks/s,  65.992 Mb/s)
  599   Tue Apr 23 19:23:42 2024 PP20:30 checks

All looks good.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_20-24-09.png
Screenshot_from_2024-04-23_20-24-09.png
Attachment 2: Screenshot_from_2024-04-23_20-24-25.png
Screenshot_from_2024-04-23_20-24-25.png
Attachment 3: Screenshot_from_2024-04-23_20-24-52.png
Screenshot_from_2024-04-23_20-24-52.png
Attachment 4: Screenshot_from_2024-04-23_20-25-35.png
Screenshot_from_2024-04-23_20-25-35.png
Attachment 5: Screenshot_from_2024-04-23_20-26-16.png
Screenshot_from_2024-04-23_20-26-16.png
Attachment 6: Screenshot_from_2024-04-23_20-27-01.png
Screenshot_from_2024-04-23_20-27-01.png
Attachment 7: Screenshot_from_2024-04-23_20-27-26.png
Screenshot_from_2024-04-23_20-27-26.png
Attachment 8: Screenshot_from_2024-04-23_20-27-53.png
Screenshot_from_2024-04-23_20-27-53.png
  598   Tue Apr 23 15:11:50 2024 TDTuesday 23 April 16:00-00:00
16.12 Analysis of data file R17_101 - attachment 1

      max deadtime c. 2% (aida02)
      no timewarps observed


16.15 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 3

      per FEE64 rate spectra (on and off spill) - attachments 4-5

      Merger etc - attachment 6


16.53 aida04 stops producing data

      DAQ STOP fails for aida04
      Restart Merger to enable aida04 to complete DAQ STOP

      aida04 rebooted and DAQ restarted OK

      data file S100_alpha/R18
Attachment 1: R17_101
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  244392363 (  344543.7 Hz)
        Other data format:   17527637 (   24710.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        632 (       0.9 Hz)
                               RESUME:        632 (       0.9 Hz)
                              SYNC100:      34642 (      48.8 Hz)
                              WR48-63:      34642 (      48.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   17457089 (   24611.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     976870 (    1377.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      709.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               16.367                0.000
  2                0.000                0.000
  3                3.744                0.000
  4                0.000                0.000
  5                0.028                0.000
  6                0.000                0.000
  7                0.009                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.004                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3247904      22147          0          0          0          0        434        434          0      21279          0     100052
  1   78164199     878648          0          0        419        419      10303      10303          0     857204          0     222655
  2    3679894    7126519          0          0          0          0       1427       1427          0    7123665          0      69753
  3   82858158    7034348          0          0        202        202      11907      11907          0    7010130          0      82407
  4    2430024    1045502          0          0          0          0        415        415          0    1044672          0      72940
  5   14446252    1404299          0          0          6          6       2074       2074          0    1400139          0      68087
  6    2997919        898          0          0          0          0        449        449          0          0          0      20036
  7   28730414       7980          0          0          4          4       3986       3986          0          0          0      64499
  8    2617862        676          0          0          0          0        338        338          0          0          0      38168
  9    3117608        872          0          0          0          0        436        436          0          0          0      15569
 10    3647866        954          0          0          0          0        477        477          0          0          0       7030
 11    3150764        810          0          0          0          0        405        405          0          0          0     129790
 12    4169396       1050          0          0          0          0        525        525          0          0          0      20235
 13    2725572        706          0          0          0          0        353        353          0          0          0      23658
 14    2606679        622          0          0          1          1        310        310          0          0          0      15170
 15    5801852       1606          0          0          0          0        803        803          0          0          0      26821
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.402s ( 1052.550 blocks/s,  65.784 Mb/s)
Attachment 2: Screenshot_from_2024-04-23_16-15-38.png
Screenshot_from_2024-04-23_16-15-38.png
Attachment 3: Screenshot_from_2024-04-23_16-16-52.png
Screenshot_from_2024-04-23_16-16-52.png
Attachment 4: Screenshot_from_2024-04-23_16-17-40.png
Screenshot_from_2024-04-23_16-17-40.png
Attachment 5: Screenshot_from_2024-04-23_16-17-25.png
Screenshot_from_2024-04-23_16-17-25.png
Attachment 6: Screenshot_from_2024-04-23_16-18-04.png
Screenshot_from_2024-04-23_16-18-04.png
Attachment 7: Screenshot_from_2024-04-23_16-22-04.png
Screenshot_from_2024-04-23_16-22-04.png
Attachment 8: Screenshot_from_2024-04-23_17-18-08.png
Screenshot_from_2024-04-23_17-18-08.png
Attachment 9: Screenshot_from_2024-04-23_17-18-34.png
Screenshot_from_2024-04-23_17-18-34.png
Attachment 10: Screenshot_from_2024-04-23_17-19-01.png
Screenshot_from_2024-04-23_17-19-01.png
Attachment 11: Screenshot_from_2024-04-23_17-19-18.png
Screenshot_from_2024-04-23_17-19-18.png
Attachment 12: Screenshot_from_2024-04-23_17-20-08.png
Screenshot_from_2024-04-23_17-20-08.png
Attachment 13: Screenshot_from_2024-04-23_17-20-47.png
Screenshot_from_2024-04-23_17-20-47.png
Attachment 14: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 15: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 16: Screenshot_from_2024-04-23_17-25-10.png
Screenshot_from_2024-04-23_17-25-10.png
Attachment 17: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 18: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 19: Screenshot_from_2024-04-23_17-29-40.png
Screenshot_from_2024-04-23_17-29-40.png
Attachment 20: Screenshot_from_2024-04-23_17-30-04.png
Screenshot_from_2024-04-23_17-30-04.png
  597   Tue Apr 23 13:15:43 2024 PP14:00 checks

All looks good.

Screenshots attached

Attachment 1: Screenshot_from_2024-04-23_14-11-33.png
Screenshot_from_2024-04-23_14-11-33.png
Attachment 2: Screenshot_from_2024-04-23_14-04-26.png
Screenshot_from_2024-04-23_14-04-26.png
Attachment 3: Screenshot_from_2024-04-23_14-04-44.png
Screenshot_from_2024-04-23_14-04-44.png
Attachment 4: Screenshot_from_2024-04-23_14-06-02.png
Screenshot_from_2024-04-23_14-06-02.png
Attachment 5: Screenshot_from_2024-04-23_14-07-07.png
Screenshot_from_2024-04-23_14-07-07.png
Attachment 6: Screenshot_from_2024-04-23_14-09-26.png
Screenshot_from_2024-04-23_14-09-26.png
Attachment 7: Screenshot_from_2024-04-23_14-10-55.png
Screenshot_from_2024-04-23_14-10-55.png
Attachment 8: Screenshot_from_2024-04-23_14-11-16.png
Screenshot_from_2024-04-23_14-11-16.png
  595   Tue Apr 23 13:08:28 2024 TDSummary of AIDA FEE64 stops
We have observed a number of AIDA FEE64s stop producing data.

Summary of dates & (approximate) times

aida03	21.4.24	08.20
aida04	22.4.24	09.14             https://elog.ph.ed.ac.uk/DESPEC/586
aida01	22.4.24	16.00
aida02	22.4.24	22.00
aida02	23.4.24	10.21
aida02	23.4.24 11.25
aida04  23.4.24 16.53?            attachment 2
aida04/aida05 25.4.24 22.28       no messages in aida04 system console
aida04  27.8.24 c.23:00
aida04  28.8.24 c. 12-13:00       power cycle
aida04  28.8.24 c. 16.00          reset per Elog 618
aida02  28.4.24 c. 22.20          reset per Elog 618, no error messages in system console
aida04  29.4.24 01.15
aida04  30.4.24 <03.30
aida02  30.4.24 <10.00


aida02 2.5.24 c. 19.30 reset


In future, it would be useful to copy system console reports to the Elog to check whether we can identify common error messages etc.


ssh pi@nnrpi1
<password>

cd logs
cat /ttyUSBx 

The current mapping of USB port # to AIDA FEE64s - attachment 1


aida02 system console c. 19.30 2.5.24

02:05:24/14:40:51|executing generic doStop
02:05:24/19:27:28|Halt (1)
02:05:24/19:27:29|Halt (2)
02:05:24/19:27:29|Halt (3)
02:05:24/19:27:30|Halt (4)
02:05:24/19:27:30|Halt (5)
02:05:24/19:27:31|Halt (6)
02:05:24/19:27:31|Halt (7)
02:05:24/19:27:32|Halt (8)
02:05:24/19:27:32|Halt (9)
02:05:24/19:27:33|Halt (10)
02:05:24/19:27:33|Halt (11)
02:05:24/19:27:34|Halt (12)
02:05:24/19:27:34|Halt (13)
02:05:24/19:27:35|Halt (14)
02:05:24/19:27:35|Halt (15)
02:05:24/19:27:36|Halt (16)
02:05:24/19:27:36|Halt (17)
02:05:24/19:27:37|Halt (18)
02:05:24/19:27:38|Halt (19)
02:05:24/19:27:38|Action has not completed
02:05:24/19:27:39|Giving up waiting
02:05:24/19:27:39|disconnect xfer stream #0 (1)
02:05:24/19:27:39|disconnect xfer stream #0 (2)
02:05:24/19:27:39|disconnect xfer stream #0 (3)
02:05:24/19:27:40|disconnect xfer stream #0 (4)
02:05:24/19:27:40|disconnect xfer stream #0 (5)
02:05:24/19:27:41|disconnect xfer stream #0 (6)
02:05:24/19:27:41|disconnect xfer stream #0 (7)
02:05:24/19:27:42|disconnect xfer stream #0 (8)
02:05:24/19:27:42|disconnect xfer stream #0 (9)
02:05:24/19:27:43|disconnect xfer stream #0 (10)
02:05:24/19:27:43|disconnect xfer stream #0 (11)
02:05:24/19:27:44|disconnect xfer stream #0 (12)
02:05:24/19:27:44|disconnect xfer stream #0 (13)
02:05:24/19:27:45|disconnect xfer stream #0 (14)
02:05:24/19:27:46|disconnect xfer stream #0 (15)
02:05:24/19:27:46|disconnect xfer stream #0 (16)
02:05:24/19:27:47|disconnect xfer stream #0 (17)
02:05:24/19:27:47|disconnect xfer stream #0 (18)
02:05:24/19:27:48|disconnect xfer stream #0 (19)
02:05:24/19:27:48|Action has not completed
02:05:24/19:27:49|Giving up waiting
02:05:24/19:27:49|executing generic Stop_Merger
02:05:24/19:27:49|executing generic Stop_DataStorageServer
02:05:24/19:27:49|completed generic doStop
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:28:01|Transfer Error - : Connection reset by peer
02:05:24/19:28:08|send() failed: 
02:05:24/19:28:08|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
02:05:24/19:28:08|do_GetState returned z=0 and 0
02:05:24/19:28:13|executing generic doGo
02:05:24/19:28:45|Initialising communication with storage server 192.168.11.99 on port 11002
02:05:24/19:28:45|Setting Transfer Block Size 65536
02:05:24/19:28:45|Setting Transfer Mode 3
02:05:24/19:28:45|Setting Overlap Mode 0
02:05:24/19:28:45|Setting Blocking Mode 0
02:05:24/19:28:45|Setting nice 0
02:05:24/19:28:45|TCP transfer library version 3.12
02:05:24/19:28:45|TCP socket send buffer was 16384 - now 221184
02:05:24/19:28:45|TCP socket receive buffer was 87380 - now 221184
02:05:24/19:28:45|TCP socket created OK - now connecting to 192.168.11.99 port 11002
02:05:24/19:28:45|Connected to 192.168.11.99 port 11002
02:05:24/19:28:45|connect xfer stream #0 (1)
02:05:24/19:28:45|Data Acquisition Statistics counters now cleared
02:05:24/19:28:46|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
02:05:24/19:28:46|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
02:05:24/19:28:46|Go (1)
02:05:24/19:28:46|Running Go_Electronics for aida02 with TSMaster aida01
02:05:24/19:28:46|ADCs Calibrated. 
02:05:24/19:28:46|Finished Go_Electronics for AIDA
02:05:24/19:28:46|executing generic Go_Merger
02:05:24/19:28:46|executing generic Go_DataStorageServer
02:05:24/19:28:46|completed generic doGo
02:05:24/19:28:46|do_GetState returned z=0 and 1
02:05:24/19:28:46|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
02:05:24/19:28:51|Also in RDOGo_Operate: Enabled waveform readout
02:05:24/19:28:51|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0201
02:05:24/19:28:51|WV:0:Buffer info. Length = 64 : Offset = 0 
02:05:24/19:28:51|WV:1:Buffer info. Length = 62 : Offset = 2 
02:05:24/19:28:51|WV:2:Buffer info. Length = 60 : Offset = 4 
02:05:24/19:28:52|WV:3:Buffer info. Length = 58 : Offset = 6 
02:05:24/19:28:52|WV:4:Buffer info. Length = 56 : Offset = 8 
02:05:24/19:28:52|WV:5:Buffer info. Length = 54 : Offset = 10 
02:05:24/19:28:52|WV:6:Buffer info. Length = 52 : Offset = 12 
02:05:24/19:28:52|WV:7:Buffer info. Length = 50 : Offset = 14 
02:05:24/19:28:52|WV:8:Buffer info. Length = 48 : Offset = 16 
02:05:24/19:28:52|WV:9:Buffer info. Length = 46 : Offset = 18 
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 1048, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|
Attachment 1: Screenshot_from_2024-04-23_14-22-04.png
Screenshot_from_2024-04-23_14-22-04.png
Attachment 2: aida04
23:04:24/11:48:07|TCP transfer library version 3.12
23:04:24/11:48:07|TCP socket send buffer was 16384 - now 221184
23:04:24/11:48:07|TCP socket receive buffer was 87380 - now 221184
23:04:24/11:48:07|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/11:48:07|Connected to 192.168.11.99 port 11004
23:04:24/11:48:07|connect xfer stream #0 (1)
23:04:24/11:48:07|Data Acquisition Statistics counters now cleared
23:04:24/11:48:07|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
23:04:24/11:48:07|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
23:04:24/11:48:07|Go (1)
23:04:24/11:48:07|Running Go_Electronics for aida04 with TSMaster aida01
23:04:24/11:48:07|ADCs Calibrated. 
23:04:24/11:48:07|Finished Go_Electronics for AIDA
23:04:24/11:48:08|executing generic Go_Merger
23:04:24/11:48:08|executing generic Go_DataStorageServer
23:04:24/11:48:08|completed generic doGo
23:04:24/11:48:08|do_GetState returned z=0 and 1
23:04:24/11:48:08|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
23:04:24/11:48:13|Also in RDOGo_Operate: Enabled waveform readout
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 104A, State machines = 0201
23:04:24/11:48:13|WV:0:Buffer info. Length = 64 : Offset = 0 
23:04:24/11:48:13|WV:1:Buffer info. Length = 62 : Offset = 2 
23:04:24/11:48:13|WV:2:Buffer info. Length = 60 : Offset = 4 
23:04:24/11:48:13|WV:3:Buffer info. Length = 58 : Offset = 6 
23:04:24/11:48:13|WV:4:Buffer info. Length = 56 : Offset = 8 
23:04:24/11:48:13|WV:5:Buffer info. Length = 54 : Offset = 10 
23:04:24/11:48:13|WV:6:Buffer info. Length = 52 : Offset = 12 
23:04:24/11:48:13|WV:7:Buffer info. Length = 50 : Offset = 14 
23:04:24/11:48:13|WV:8:Buffer info. Length = 48 : Offset = 16 
23:04:24/11:48:13|WV:9:Buffer info. Length = 46 : Offset = 18 
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0188, State machines = 0201
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|executing generic doStop
23:04:24/16:58:30|Halt (1)
23:04:24/16:58:31|Halt (2)
23:04:24/16:58:32|Halt (3)
23:04:24/16:58:32|Halt (4)
23:04:24/16:58:33|Halt (5)
23:04:24/16:58:33|Halt (6)
23:04:24/16:58:34|Halt (7)
23:04:24/16:58:35|Halt (8)
23:04:24/16:58:35|Halt (9)
23:04:24/16:58:36|Halt (10)
23:04:24/16:58:36|Halt (11)
23:04:24/16:58:37|Halt (12)
23:04:24/16:58:37|Halt (13)
23:04:24/16:58:38|Halt (14)
23:04:24/16:58:39|Halt (15)
23:04:24/16:58:39|Halt (16)
23:04:24/16:58:40|Halt (17)
23:04:24/16:58:40|Halt (18)
23:04:24/16:58:41|Halt (19)
23:04:24/16:58:41|Action has not completed
23:04:24/16:58:42|Giving up waiting
23:04:24/16:58:42|disconnect xfer stream #0 (1)
23:04:24/16:58:42|disconnect xfer stream #0 (2)
23:04:24/16:58:43|disconnect xfer stream #0 (3)
23:04:24/16:58:43|disconnect xfer stream #0 (4)
23:04:24/16:58:44|disconnect xfer stream #0 (5)
23:04:24/16:58:44|disconnect xfer stream #0 (6)
23:04:24/16:58:45|disconnect xfer stream #0 (7)
23:04:24/16:58:45|disconnect xfer stream #0 (8)
23:04:24/16:58:46|disconnect xfer stream #0 (9)
23:04:24/16:58:47|disconnect xfer stream #0 (10)
23:04:24/16:58:47|disconnect xfer stream #0 (11)
23:04:24/16:58:48|disconnect xfer stream #0 (12)
23:04:24/16:58:48|disconnect xfer stream #0 (13)
23:04:24/16:58:49|disconnect xfer stream #0 (14)
23:04:24/16:58:49|disconnect xfer stream #0 (15)
23:04:24/16:58:50|disconnect xfer stream #0 (16)
23:04:24/16:58:51|disconnect xfer stream #0 (17)
23:04:24/16:58:51|disconnect xfer stream #0 (18)
23:04:24/16:58:52|disconnect xfer stream #0 (19)
23:04:24/16:58:52|Action has not completed
23:04:24/16:58:53|Giving up waiting
23:04:24/16:58:53|executing generic Stop_Merger
23:04:24/16:58:53|executing generic Stop_DataStorageServer
23:04:24/16:58:53|completed generic doStop
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:59:06|Transfer Error - : Connection reset by peer
23:04:24/16:59:15|send() failed: 
23:04:24/16:59:15|TCP transfer library version 3.12
23:04:24/16:59:15|TCP socket send buffer was 16384 - now 221184
23:04:24/16:59:15|TCP socket receive buffer was 87380 - now 221184
23:04:24/16:59:15|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/16:59:15|Transfer Error - : Connection refused
23:04:24/16:59:15|connect() failed:  
23:04:24/16:59:15|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
23:04:24/16:59:15|do_GetState returned z=0 and 0
23:04:24/16:59:18|do_GetState returned z=0 and 0
23:04:24/17:00:09|do_GetState returned z=0 and 0
23:04:24/17:00:11|do_GetState returned z=0 and 0
23:04:24/17:00:19|do_GetState returned z=0 and 0
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
23:04:24/17:00:43|[  OK  ]
23:04:24/17:00:43|Stopping rpcbind: [  OK  ]
23:04:24/17:00:45|Shutting down kernel logger: [  OK  ]
23:04:24/17:00:47|Shutting down system logger: [  OK  ]
23:04:24/17:00:47|Stopping xinetd: [  OK  ]
23:04:24/17:00:47|[  OK  ]
23:04:24/17:00:47|Sending all processes the TERM signal... xaida: release:
23:04:24/17:00:48|
23:04:24/17:00:48|AIDAMEM: aidamem_release:
23:04:24/17:00:48|xaida: release:
23:04:24/17:00:48|Sending all processes the KILL signal... 
23:04:24/17:00:53|Saving random seed:  
23:04:24/17:00:53|Syncing hardware clock to systxaida: open:
23:04:24/17:00:53|emxaida: release:
23:04:24/17:00:53| time xaida: open:
23:04:24/17:00:53|xaida: ioctl: called with command 536899587
23:04:24/17:00:53|xaida: ioctl: called with command 1076129801
23:04:24/17:00:53|RTCxaida: release:
23:04:24/17:00:53|_RD_TIME: Inappropriate ioctl for device
23:04:24/17:00:53|ioctl() to /dev/rtc to read the time failed.
23:04:24/17:00:53|
23:04:24/17:00:53|Please stand by while rebooting the system...
Sj5õ24/17:00:54|Restarting system.
23:04:24/17:00:56|U×®­Ë
                        )	$T,W¤']Z¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:00:56|
Finished copying zImage to RAM 
23:04:24/17:00:57|
Found 0 errors checking kernel image
23:04:24/17:00:58|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
23:04:24/17:00:58|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
23:04:24/17:00:58|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
23:04:24/17:00:58|SPIconfig done now to set up the LMK3200 registers
23:04:24/17:00:58|LMK #0 : regInit[0]=0x80000000
23:04:24/17:00:58|LMK #0 : regInit[1]=0x10070600
23:04:24/17:00:58|LMK #0 : regInit[2]=0x60601
23:04:24/17:00:59|LMK #0 : regInit[3]=0x60602
23:04:24/17:00:59|LMK #0 : regInit[4]=0x60603
23:04:24/17:00:59|LMK #0 : regInit[5]=0x70624
23:04:24/17:00:59|LMK #0 : regInit[6]=0x70605
23:04:24/17:00:59|LMK #0 : regInit[7]=0x70606
23:04:24/17:00:59|LMK #0 : regInit[8]=0x70627
23:04:24/17:00:59|LMK #0 : regInit[9]=0x10000908
23:04:24/17:00:59|LMK #0 : regInit[10]=0xA0022A09
23:04:24/17:00:59|LMK #0 : regInit[11]=0x82800B
23:04:24/17:00:59|LMK #0 : regInit[12]=0x28C800D
23:04:24/17:00:59|LMK #0 : regInit[13]=0x830020E
23:04:24/17:00:59|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
23:04:24/17:00:59|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
23:04:24/17:00:59|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
23:04:24/17:01:00|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
23:04:24/17:01:00|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
23:04:24/17:01:03|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xa0 0x71 
23:04:24/17:01:09|Using Xilinx Virtex440 machine description
23:04:24/17:01:10|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
23:04:24/17:01:10|Zone PFN ranges:
23:04:24/17:01:10|  DMA      0x00000000 -> 0x00007000
23:04:24/17:01:10|  Normal   0x00007000 -> 0x00007000
23:04:24/17:01:10|Movable zone start PFN for each node
23:04:24/17:01:10|early_node_map[1] active PFN ranges
23:04:24/17:01:10|    0: 0x00000000 -> 0x00007000
23:04:24/17:01:10|MMU: Allocated 1088 bytes of context maps for 255 contexts
23:04:24/17:01:10|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
23:04:24/17:01:10|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
23:04:24/17:01:10|PID hash table entries: 512 (order: 9, 2048 bytes)
23:04:24/17:01:10|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
23:04:24/17:01:11|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
23:04:24/17:01:11|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
23:04:24/17:01:11|Kernel virtual memory layout:
23:04:24/17:01:11|  * 0xffffe000..0xfffff000  : fixmap
23:04:24/17:01:11|  * 0xfde00000..0xfe000000  : consistent mem
23:04:24/17:01:11|  * 0xfde00000..0xfde00000  : early ioremap
23:04:24/17:01:11|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
23:04:24/17:01:11|NR_IRQS:512
23:04:24/17:01:11|clocksource: timebase mult[a00000] shift[22] registered
23:04:24/17:01:11|Console: colour dummy device 80x25
23:04:24/17:01:11|Mount-cache hash table entries: 512
23:04:24/17:01:11|NET: Registered protocol family 16
23:04:24/17:01:11|PCI: Probing PCI hardware
23:04:24/17:01:11|bio: create slab <bio-0> at 0
23:04:24/17:01:11|NET: Registered protocol family 2
23:04:24/17:01:11|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
23:04:24/17:01:11|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
23:04:24/17:01:11|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
23:04:24/17:01:11|TCP: Hash tables configured (established 4096 bind 4096)
23:04:24/17:01:11|TCP reno registered
23:04:24/17:01:12|NET: Registered protocol family 1
23:04:24/17:01:12|ROMFS MTD (C) 2007 Red Hat, Inc.
23:04:24/17:01:12|msgmni has been set to 214
23:04:24/17:01:12|io scheduler noop registered
23:04:24/17:01:12|io scheduler anticipatory registered
23:04:24/17:01:12|io scheduler deadline registered
23:04:24/17:01:12|io scheduler cfq registered (default)
23:04:24/17:01:12|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
23:04:24/17:01:12|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
23:04:24/17:01:12|console [ttyS0] enabled
23:04:24/17:01:12|brd: module loaded
23:04:24/17:01:12|loop: module loaded
23:04:24/17:01:12|Device Tree Probing 'ethernet'
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:a0:71
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
23:04:24/17:01:12|XLlTemac: DCR address: 0x80
23:04:24/17:01:12|XLlTemac: buffer descriptor size: 32768 (0x8000)
23:04:24/17:01:12|XLlTemac: Allocating DMA descriptors with kmalloc
23:04:24/17:01:12|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
23:04:24/17:01:12|XTemac: PHY detected at address 7.
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
23:04:24/17:01:13|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13|Using buffer write method
23:04:24/17:01:13|cfi_cmdset_0001: Erase suspend on write enabled
23:04:24/17:01:13|cmdlinepart partition parsing not available
23:04:24/17:01:13|RedBoot partition parsing not available
23:04:24/17:01:13|Creating 5 MTD partitions on "fc000000.flash":
23:04:24/17:01:13|0x000000000000-0x000000500000 : "golden_firmware"
23:04:24/17:01:13|0x000000500000-0x000000800000 : "golden_kernel"
23:04:24/17:01:13|0x000000800000-0x000000d00000 : "user_firmware"
23:04:24/17:01:13|0x000000d00000-0x000000fe0000 : "user_kernel"
23:04:24/17:01:13|0x000000fe0000-0x000001000000 : "env_variables"
23:04:24/17:01:13|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
23:04:24/17:01:13|SPI: XIlinx spi: bus number now 32766
23:04:24/17:01:13|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
23:04:24/17:01:14|SPI: XIlinx spi: bus number now 32765
23:04:24/17:01:14|mice: PS/2 mouse device common for all mice
23:04:24/17:01:14|Device Tree Probing 'i2c'
23:04:24/17:01:14| #0 at 0x81600000 mapped to 0xD1030000, irq=22
23:04:24/17:01:14|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
23:04:24/17:01:14|TCP cubic registered
23:04:24/17:01:14|NET: Registered protocol family 17
23:04:24/17:01:14|RPC: Registered udp transport module.
23:04:24/17:01:14|RPC: Registered tcp transport module.
23:04:24/17:01:14|eth0: XLlTemac: Options: 0x3fa
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
23:04:24/17:01:15|eth0: XLlTemac: speed set to 1000Mb/s
23:04:24/17:01:17|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
23:04:24/17:01:17|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
23:04:24/17:01:17|Sending DHCP requests ., OK
23:04:24/17:01:18|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
23:04:24/17:01:18|IP-Config: Complete:
23:04:24/17:01:18|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
23:04:24/17:01:18|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
23:04:24/17:01:18|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
23:04:24/17:01:18|Looking up port of RPC 100003/2 on 192.168.11.99
23:04:24/17:01:18|Looking up port of RPC 100005/1 on 192.168.11.99
23:04:24/17:01:18|VFS: Mounted root (nfs filesystem) on device 0:12.
23:04:24/17:01:18|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
23:04:24/17:01:19|Starting sysinit...
23:04:24/17:01:19|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
23:04:24/17:01:20|		Press 'I' to enter interactive startup.
23:04:24/17:01:20|Setting clock  (utc): Tue Apr 23 16:01:20 BST 2024 [  OK  ]
23:04:24/17:01:20|Building the cache [  OK  ]
23:04:24/17:01:20|Setting hostname aida04:  [  OK  ]
23:04:24/17:01:21|Mounting local filesystems:  [  OK  ]
23:04:24/17:01:22|Enabling /etc/fstab swaps:  [  OK  ]
23:04:24/17:01:24|Finishing sysinit...
INIT: Entering runlevel: 3
... 257 more lines ...
  594   Tue Apr 23 11:09:00 2024 PP12:00 checks

Everything is OK now.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_12-01-49.png
Screenshot_from_2024-04-23_12-01-49.png
Attachment 2: Screenshot_from_2024-04-23_12-02-24.png
Screenshot_from_2024-04-23_12-02-24.png
Attachment 3: Screenshot_from_2024-04-23_12-03-36.png
Screenshot_from_2024-04-23_12-03-36.png
Attachment 4: Screenshot_from_2024-04-23_12-04-05.png
Screenshot_from_2024-04-23_12-04-05.png
Attachment 5: Screenshot_from_2024-04-23_12-05-52.png
Screenshot_from_2024-04-23_12-05-52.png
Attachment 6: Screenshot_from_2024-04-23_12-06-42.png
Screenshot_from_2024-04-23_12-06-42.png
Attachment 7: Screenshot_from_2024-04-23_12-07-41.png
Screenshot_from_2024-04-23_12-07-41.png
Attachment 8: Screenshot_from_2024-04-23_12-08-32.png
Screenshot_from_2024-04-23_12-08-32.png
  593   Tue Apr 23 10:52:39 2024 PPAida02 down and back

We lost aida02 at about 11:25 again. Nick fixed it about 11:50.

It is now back in the DAQ.

  592   Tue Apr 23 10:10:31 2024 PPaida02 is back

Nick rebooted AIDA and aida02 came back at around 10:55.

AIDA was out of the DAQ from about 10:30 until 10:55

  591   Tue Apr 23 09:21:01 2024 PP10:20 checks - aida02 is down

Aida02 is down. Contacted Nick.

The rest looks stable, see screenshots.

10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14

Go > All 16 writing to merger, looks OK

 

aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again

Attachment 1: Screenshot_from_2024-04-23_10-08-54.png
Screenshot_from_2024-04-23_10-08-54.png
Attachment 2: Screenshot_from_2024-04-23_10-13-23.png
Screenshot_from_2024-04-23_10-13-23.png
Attachment 3: Screenshot_from_2024-04-23_10-15-28.png
Screenshot_from_2024-04-23_10-15-28.png
Attachment 4: Screenshot_from_2024-04-23_10-16-23.png
Screenshot_from_2024-04-23_10-16-23.png
Attachment 5: Screenshot_from_2024-04-23_10-16-47.png
Screenshot_from_2024-04-23_10-16-47.png
Attachment 6: Screenshot_from_2024-04-23_10-17-47.png
Screenshot_from_2024-04-23_10-17-47.png
Attachment 7: Screenshot_from_2024-04-23_10-19-30.png
Screenshot_from_2024-04-23_10-19-30.png
Attachment 8: Screenshot_from_2024-04-23_10-20-20.png
Screenshot_from_2024-04-23_10-20-20.png
  590   Tue Apr 23 07:27:18 2024 PP08:00-16:00 Tuesday 24 April shift

Checks 08:00-08:30

Screenshots are attached and everything looks OK.

Attachment 1: Screenshot_from_2024-04-23_08-10-21.png
Screenshot_from_2024-04-23_08-10-21.png
Attachment 2: Screenshot_from_2024-04-23_08-13-20.png
Screenshot_from_2024-04-23_08-13-20.png
Attachment 3: Screenshot_from_2024-04-23_08-15-30.png
Screenshot_from_2024-04-23_08-15-30.png
Attachment 4: Screenshot_from_2024-04-23_08-18-28.png
Screenshot_from_2024-04-23_08-18-28.png
Attachment 5: Screenshot_from_2024-04-23_08-23-07.png
Screenshot_from_2024-04-23_08-23-07.png
Attachment 6: Screenshot_from_2024-04-23_08-26-31.png
Screenshot_from_2024-04-23_08-26-31.png
Attachment 7: Screenshot_from_2024-04-23_08-35-31.png
Screenshot_from_2024-04-23_08-35-31.png
Attachment 8: Screenshot_from_2024-04-23_08-39-43.png
Screenshot_from_2024-04-23_08-39-43.png
  589   Mon Apr 22 22:35:59 2024 TD00.00-08.00 Tuesday 22 April
23.36 Analysis of data file R17_34 - attachment 1
      max deadtime c. 1% (aida02)
      no timewarps

23.38 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 3
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

01.03 DSSSD bias & leakage current OK - attachment 7

      FEE64 temperatures OK - attachment 8
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 9

      per FEE64 Rate spectra - attachment 10

      Merger etc - attachment 11

03.06 DSSSD bias & leakage current OK - attachment 12

      FEE64 temperatures OK - attachment 13
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 14

      per FEE64 Rate spectra - attachment 15

      Merger etc - attachment 16

05.04 DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Merger etc - attachment 21

07.29 DSSSD bias & leakage current OK - attachment 22

      FEE64 temperatures OK - attachment 23
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 24

      per FEE64 Rate spectra - attachment 25

      Merger etc - attachment 26
Attachment 1: R17_34
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  242086746 (  241481.1 Hz)
        Other data format:   19833254 (   19783.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        347 (       0.3 Hz)
                               RESUME:        347 (       0.3 Hz)
                              SYNC100:      35735 (      35.6 Hz)
                              WR48-63:      35735 (      35.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   19761090 (   19711.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     676877 (     675.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     1002.508 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                7.276                0.000
  2                0.000                0.000
  3                3.384                0.000
  4                0.000                0.000
  5                0.005                0.000
  6                0.000                0.000
  7                0.131                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.020                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2611636      30762          0          0          0          0        342        342          0      30078          0      64523
  1   15105593     133663          0          0        117        117       1949       1949          0     129531          0      31773
  2    4737354    9053995          0          0          0          0       1811       1811          0    9050373          0      45719
  3   93081574    8973436          0          0        187        187      14170      14170          0    8944722          0      56135
  4    2149658     687339          0          0          0          0        339        339          0     686661          0      47171
  5   16270251     924233          0          0          3          3       2251       2251          0     919725          0      42612
  6    4771409       1270          0          0          0          0        635        635          0          0          0      12858
  7   64665607      18074          0          0         39         39       8998       8998          0          0          0     197755
  8    1902284        428          0          0          0          0        214        214          0          0          0      25268
  9    4387089       1362          0          0          0          0        681        681          0          0          0      10256
 10    8523582       2388          0          0          0          0       1194       1194          0          0          0       4627
 11    2287700        576          0          0          0          0        288        288          0          0          0      83305
 12    4391198       1112          0          0          0          0        556        556          0          0          0      12633
 13    3103598        860          0          0          0          0        430        430          0          0          0      15208
 14    1965500        506          0          0          0          0        253        253          0          0          0      10217
 15   12132713       3250          0          0          1          1       1624       1624          0          0          0      16817
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.539s ( 1047.838 blocks/s,  65.490 Mb/s)
Attachment 2: Screenshot_from_2024-04-22_23-38-08.png
Screenshot_from_2024-04-22_23-38-08.png
Attachment 3: Screenshot_from_2024-04-22_23-39-17.png
Screenshot_from_2024-04-22_23-39-17.png
Attachment 4: Screenshot_from_2024-04-22_23-39-44.png
Screenshot_from_2024-04-22_23-39-44.png
Attachment 5: Screenshot_from_2024-04-22_23-40-27.png
Screenshot_from_2024-04-22_23-40-27.png
Attachment 6: Screenshot_from_2024-04-22_23-40-54.png
Screenshot_from_2024-04-22_23-40-54.png
Attachment 7: Screenshot_from_2024-04-23_01-04-32.png
Screenshot_from_2024-04-23_01-04-32.png
Attachment 8: Screenshot_from_2024-04-23_01-05-40.png
Screenshot_from_2024-04-23_01-05-40.png
Attachment 9: Screenshot_from_2024-04-23_01-06-12.png
Screenshot_from_2024-04-23_01-06-12.png
Attachment 10: Screenshot_from_2024-04-23_01-06-40.png
Screenshot_from_2024-04-23_01-06-40.png
Attachment 11: Screenshot_from_2024-04-23_01-07-18.png
Screenshot_from_2024-04-23_01-07-18.png
Attachment 12: Screenshot_from_2024-04-23_03-06-28.png
Screenshot_from_2024-04-23_03-06-28.png
Attachment 13: Screenshot_from_2024-04-23_03-07-02.png
Screenshot_from_2024-04-23_03-07-02.png
Attachment 14: Screenshot_from_2024-04-23_03-07-35.png
Screenshot_from_2024-04-23_03-07-35.png
Attachment 15: Screenshot_from_2024-04-23_03-08-09.png
Screenshot_from_2024-04-23_03-08-09.png
Attachment 16: Screenshot_from_2024-04-23_03-08-43.png
Screenshot_from_2024-04-23_03-08-43.png
Attachment 17: Screenshot_from_2024-04-23_05-02-11.png
Screenshot_from_2024-04-23_05-02-11.png
Attachment 18: Screenshot_from_2024-04-23_05-03-10.png
Screenshot_from_2024-04-23_05-03-10.png
Attachment 19: Screenshot_from_2024-04-23_05-03-57.png
Screenshot_from_2024-04-23_05-03-57.png
Attachment 20: Screenshot_from_2024-04-23_05-04-37.png
Screenshot_from_2024-04-23_05-04-37.png
Attachment 21: Screenshot_from_2024-04-23_05-05-13.png
Screenshot_from_2024-04-23_05-05-13.png
Attachment 22: Screenshot_from_2024-04-23_07-29-15.png
Screenshot_from_2024-04-23_07-29-15.png
Attachment 23: Screenshot_from_2024-04-23_07-29-59.png
Screenshot_from_2024-04-23_07-29-59.png
Attachment 24: Screenshot_from_2024-04-23_07-30-27.png
Screenshot_from_2024-04-23_07-30-27.png
Attachment 25: Screenshot_from_2024-04-23_07-30-54.png
Screenshot_from_2024-04-23_07-30-54.png
Attachment 26: Screenshot_from_2024-04-23_07-31-16.png
Screenshot_from_2024-04-23_07-31-16.png
  588   Mon Apr 22 15:17:07 2024 DSJ16.00-0.00 22/04/24

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

 

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

 

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Attachment 1: Screenshot_from_2024-04-22_16-16-30.png
Screenshot_from_2024-04-22_16-16-30.png
Attachment 2: Screenshot_from_2024-04-22_16-14-31.png
Screenshot_from_2024-04-22_16-14-31.png
Attachment 3: Screenshot_from_2024-04-22_16-09-16.png
Screenshot_from_2024-04-22_16-09-16.png
Attachment 4: Screenshot_from_2024-04-22_16-05-22.png
Screenshot_from_2024-04-22_16-05-22.png
Attachment 5: Screenshot_from_2024-04-22_16-33-34.png
Screenshot_from_2024-04-22_16-33-34.png
Attachment 6: Screenshot_from_2024-04-22_16-32-54.png
Screenshot_from_2024-04-22_16-32-54.png
Attachment 7: Screenshot_from_2024-04-22_16-32-04.png
Screenshot_from_2024-04-22_16-32-04.png
Attachment 8: Screenshot_from_2024-04-22_16-31-03.png
Screenshot_from_2024-04-22_16-31-03.png
Attachment 9: Screenshot_from_2024-04-22_16-30-00.png
Screenshot_from_2024-04-22_16-30-00.png
Attachment 10: Screenshot_from_2024-04-22_16-29-03.png
Screenshot_from_2024-04-22_16-29-03.png
Attachment 11: Screenshot_from_2024-04-22_16-27-01.png
Screenshot_from_2024-04-22_16-27-01.png
Attachment 12: Screenshot_from_2024-04-22_16-25-30.png
Screenshot_from_2024-04-22_16-25-30.png
Attachment 13: Screenshot_from_2024-04-22_19-00-04.png
Screenshot_from_2024-04-22_19-00-04.png
Attachment 14: Screenshot_from_2024-04-22_18-59-25.png
Screenshot_from_2024-04-22_18-59-25.png
Attachment 15: Screenshot_from_2024-04-22_18-58-21.png
Screenshot_from_2024-04-22_18-58-21.png
Attachment 16: Screenshot_from_2024-04-22_18-56-34.png
Screenshot_from_2024-04-22_18-56-34.png
Attachment 17: Screenshot_from_2024-04-22_20-46-50.png
Screenshot_from_2024-04-22_20-46-50.png
Attachment 18: Screenshot_from_2024-04-22_20-45-57.png
Screenshot_from_2024-04-22_20-45-57.png
Attachment 19: Screenshot_from_2024-04-22_20-45-10.png
Screenshot_from_2024-04-22_20-45-10.png
Attachment 20: Screenshot_from_2024-04-22_20-44-41.png
Screenshot_from_2024-04-22_20-44-41.png
Attachment 21: Screenshot_from_2024-04-22_22-00-30.png
Screenshot_from_2024-04-22_22-00-30.png
Attachment 22: Screenshot_from_2024-04-22_22-53-11.png
Screenshot_from_2024-04-22_22-53-11.png
Attachment 23: Screenshot_from_2024-04-22_22-52-19.png
Screenshot_from_2024-04-22_22-52-19.png
Attachment 24: Screenshot_from_2024-04-22_22-50-52.png
Screenshot_from_2024-04-22_22-50-52.png
Attachment 25: Screenshot_from_2024-04-22_22-49-07.png
Screenshot_from_2024-04-22_22-49-07.png
Attachment 26: Screenshot_from_2024-04-22_22-48-31.png
Screenshot_from_2024-04-22_22-48-31.png
  587   Mon Apr 22 11:43:55 2024 TD21-22.4.24 overnight MIDASsort online sort
Attachment 1 - per DSSSD LEC e_p versus e_n (20keV/channel)

Attachment 2 - per DSSSD HEC m_p versus m_n 

Attachment 3 - per DSSSD HEC x versus y (I really do need to sort out the mapping!)

Attachment 4 - per DSSSD HEC e_p versus e_n (20MeV/channel)

Attachment 5 - per DSSSD per pixel HEC-HEC time (4.096us/channel)

Attachment 6 - per DSSSD implant/decay event rates (262.144us/channel)
               showing beam spill cycle

Attachment 7 - colours/contour levels for all 2D spectra
Attachment 1: Screenshot_from_2024-04-22_07-58-12.png
Screenshot_from_2024-04-22_07-58-12.png
Attachment 2: Screenshot_from_2024-04-22_07-59-15.png
Screenshot_from_2024-04-22_07-59-15.png
Attachment 3: Screenshot_from_2024-04-22_08-00-37.png
Screenshot_from_2024-04-22_08-00-37.png
Attachment 4: Screenshot_from_2024-04-22_08-01-08.png
Screenshot_from_2024-04-22_08-01-08.png
Attachment 5: Screenshot_from_2024-04-22_08-02-03.png
Screenshot_from_2024-04-22_08-02-03.png
Attachment 6: Screenshot_from_2024-04-22_08-03-25.png
Screenshot_from_2024-04-22_08-03-25.png
Attachment 7: Screenshot_from_2024-04-22_12-48-18.png
Screenshot_from_2024-04-22_12-48-18.png
  586   Mon Apr 22 09:03:54 2024 TDReport: aida04 stops producing data
09.14 aida04 stopped producing data - system console output - attachment 1

It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.

Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?

rebooted aida04

DAQ RESET/SETUP etc cycle for all FEE64s

10.03 successful restart 
      current data file R15
Attachment 1: Screenshot_from_2024-04-22_09-49-30.png
Screenshot_from_2024-04-22_09-49-30.png
  585   Sun Apr 21 23:20:45 2024 TDMonday 22 April
00.15 Data link to MBS was closed by MBS - reason(s) unknown
      NH restarted MBS receiver and data transfer restarted

01.34 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      Merger, TapeServer etc - attachment 4

04.36 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      Merger, TapeServer etc - attachment 8

07.54 data storage to local disk enabled
      file aida-gsi:/TapeData/S100_alpha/R14_

09.36 DAQ continues, current data file R14_46

      168Er setting

      DSSSD bias & leakage current OK - attachments 9 & 10
      leakage current DSSSD#1 ramping and cycling with beam spills

      FEE64 temperatures OK - attachment 11

      System wide checks OK *except* WR/FPGA errors - attachments 12-13

      WR timestamps - attachment 14
       aida04 out of seq

      ADC data item stats - attachment 15
       high rates observed all FEE64s beam on, beam off rates OK

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.L spectra - attachments 17-18

      per FEE64 1.8.W spectra - attachments 19-20

      per FEE^4 1.8.H spectra - attachments 21-22
   
      Merger, TapeServer etc - attachments 23-25
       data rates to c. 10Mb/s beam on


09.55 aida04 rebooted - https://elog.ph.ed.ac.uk/DESPEC/586

10.01 DAQ restarted
      data file R15

      ADC data item stats - attachments 26-27
       large variation in rates beam off/on

c. 10.30 beam off (for new FRS setting?)


11.25 analysis of data files R14_19 (beam ON) and R15_24 (beam OFF) - attachments 28-29

      For beam ON max deadtime c. 11% (aida04), beam OFF max deadtime c. 0.1% (aida04)

      Zero timewarps observed

      Notice large # of PAUSE/RESUME data items for aida03 & aida04 presumably due to additional overhead of processing the time machine scalers 

12.35 Beam off

      DSSSD bias & leakage current OK - attachment 30
       Can observe beam spill structure in DSSSD leakage currents - now slowly recovering with beam off

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachment 32

      Merger, TapeServer etc - attachment 33

12.55 New ASIC settings 2024Apr22-12.55.38
       *all* discs disabled to reduce load on aida04
       slow comparator 0x14

      BNC PB-5 settings (since yesterday evening)

      Amplitude 10.0V
      Attenuation x10
      tau_d 1ms
      Frequency 2Hz
      Tail pulse
      Polarity +


13.46 all histograms & stats zero'd

13.48 Beam off
      per FEE64 Rate spectra - attachment 34
       2Hz pulser to p+n FEE64s

      per FEE64 1.8.W spectra - attachments 35-36
       p+n FEE64 threshold = 6000 to capture pulser events
       n+n FEE64 threshold = 8000 to capture noise

      per p+n FEE64 1.8.L spectra - attachments 37-38
       common x/y-scale - peak height proxy for peak width
       aida09 pulser peak width 57 ch FWHM ~40keV FWHM ( 5 x sigma ~ 85keV )

14.28 168Eu setting (check only - setup still in progress)
Attachment 1: Screenshot_from_2024-04-22_01-32-28.png
Screenshot_from_2024-04-22_01-32-28.png
Attachment 2: Screenshot_from_2024-04-22_01-32-48.png
Screenshot_from_2024-04-22_01-32-48.png
Attachment 3: Screenshot_from_2024-04-22_01-33-05.png
Screenshot_from_2024-04-22_01-33-05.png
Attachment 4: Screenshot_from_2024-04-22_01-34-06.png
Screenshot_from_2024-04-22_01-34-06.png
Attachment 5: Screenshot_from_2024-04-22_05-12-23.png
Screenshot_from_2024-04-22_05-12-23.png
Attachment 6: Screenshot_from_2024-04-22_05-13-16.png
Screenshot_from_2024-04-22_05-13-16.png
Attachment 7: Screenshot_from_2024-04-22_05-14-07.png
Screenshot_from_2024-04-22_05-14-07.png
Attachment 8: Screenshot_from_2024-04-22_05-15-01.png
Screenshot_from_2024-04-22_05-15-01.png
Attachment 9: Screenshot_from_2024-04-22_09-27-20.png
Screenshot_from_2024-04-22_09-27-20.png
Attachment 10: Screenshot_from_2024-04-22_09-37-32.png
Screenshot_from_2024-04-22_09-37-32.png
Attachment 11: Screenshot_from_2024-04-22_09-27-38.png
Screenshot_from_2024-04-22_09-27-38.png
Attachment 12: Screenshot_from_2024-04-22_09-27-54.png
Screenshot_from_2024-04-22_09-27-54.png
Attachment 13: Screenshot_from_2024-04-22_09-27-54.png
Screenshot_from_2024-04-22_09-27-54.png
Attachment 14: Screenshot_from_2024-04-22_09-28-29.png
Screenshot_from_2024-04-22_09-28-29.png
Attachment 15: Screenshot_from_2024-04-22_09-28-43.png
Screenshot_from_2024-04-22_09-28-43.png
Attachment 16: Screenshot_from_2024-04-22_09-29-28.png
Screenshot_from_2024-04-22_09-29-28.png
Attachment 17: Screenshot_from_2024-04-22_09-30-17.png
Screenshot_from_2024-04-22_09-30-17.png
Attachment 18: Screenshot_from_2024-04-22_09-30-48.png
Screenshot_from_2024-04-22_09-30-48.png
Attachment 19: Screenshot_from_2024-04-22_09-31-45.png
Screenshot_from_2024-04-22_09-31-45.png
Attachment 20: Screenshot_from_2024-04-22_09-32-00.png
Screenshot_from_2024-04-22_09-32-00.png
Attachment 21: Screenshot_from_2024-04-22_09-34-15.png
Screenshot_from_2024-04-22_09-34-15.png
Attachment 22: Screenshot_from_2024-04-22_09-35-55.png
Screenshot_from_2024-04-22_09-35-55.png
Attachment 23: Screenshot_from_2024-04-22_09-36-14.png
Screenshot_from_2024-04-22_09-36-14.png
Attachment 24: Screenshot_from_2024-04-22_09-36-26.png
Screenshot_from_2024-04-22_09-36-26.png
Attachment 25: Screenshot_from_2024-04-22_09-36-37.png
Screenshot_from_2024-04-22_09-36-37.png
Attachment 26: Screenshot_from_2024-04-22_10-06-16.png
Screenshot_from_2024-04-22_10-06-16.png
Attachment 27: Screenshot_from_2024-04-22_10-06-32.png
Screenshot_from_2024-04-22_10-06-32.png
Attachment 28: R14_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  247983505 (  854821.8 Hz)
        Other data format:   13936495 (   48040.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2020 (       7.0 Hz)
                               RESUME:       2018 (       7.0 Hz)
                              SYNC100:      33080 (     114.0 Hz)
                              WR48-63:      33080 (     114.0 Hz)
                           FEE64 disc:    4068663 (   14025.1 Hz)
                             MBS info:    9797634 (   33773.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:    1220258 (    4206.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.100 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.648                0.000
  1               16.396                0.000
  2               16.224                0.000
  3               32.946                0.000
  4                4.383                0.000
  5               16.546                0.000
  6                0.000                0.000
  7               17.903                0.000
  8                0.098                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.093                0.000
 12                0.128                0.000
 13                0.000                0.000
 14                6.899                0.000
 15                0.006                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16217774     546956          0          0         43         43       2226       2226     533739       8679          0     243502
  1   35997019     655408          0          0        266        266       4741       4741     645391          3          0     273982
  2   19522166    4283326          0          0        212        211       2914       2914     609647    3667428          0     121715
  3   45252368    3863960          0          0        711        711       6334       6334     501021    3348849          0      85285
  4   13017873    1445769          0          0        136        136       1769       1769     113373    1328586          0      34444
  5   28657824    1675874          0          0        252        251       3721       3721     223840    1444089          0      62230
  6    2024666     103995          0          0          1          1        248        248     103497          0          0      18668
  7   35214745     192536          0          0        242        242       4568       4568     182916          0          0      92814
  8    9901593     241512          0          0          8          8       1198       1198     239100          0          0      66118
  9    1636292      52204          0          0          0          0        256        256      51692          0          0       9281
 10    1524839      18248          0          0          0          0        211        211      17826          0          0       2366
 11    7330756     257048          0          0         12         12        910        910     255204          0          0      83995
 12    5748840     120224          0          0         17         17        694        694     118802          0          0      35149
 13    2639782     121147          0          0          0          0        330        330     120487          0          0      30525
 14   17865155     154024          0          0        117        117       2280       2280     149230          0          0      12092
 15    5431813     204264          0          0          3          3        680        680     202898          0          0      48092
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.855s ( 1071.830 blocks/s,  66.989 Mb/s)
Attachment 29: R15_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  228207643 (  129261.6 Hz)
        Other data format:   33712360 (   19095.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        547 (       0.3 Hz)
                               RESUME:        547 (       0.3 Hz)
                              SYNC100:      38576 (      21.9 Hz)
                              WR48-63:      38576 (      21.9 Hz)
                           FEE64 disc:    2636338 (    1493.3 Hz)
                             MBS info:   30997776 (   17557.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     110998 (      62.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     1765.471 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                1.254                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.032                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     716460      53177          0          0          0          0        106        106          0      52965          0          0
  1    6234512      54773          0          0          0          0        905        905          1      52962          0          1
  2    1818908   15542833          0          0          0          0       2263       2263      44321   15493986          0          0
  3  117796308   17921646          0          0        542        542      20384      20384    2481931   15397863          0          3
  4     548813        118          0          0          0          0         59         59          0          0          0          0
  5    7623899       2086          0          0          0          0       1043       1043          0          0          0          0
  6    3790120       8410          0          0          0          0        635        635       7140          0          0          0
  7   58887930      93835          0          0          5          5       8712       8712      76401          0          0     110890
  8     284598       7007          0          0          0          0         30         30       6947          0          0          2
  9    5466145       1638          0          0          0          0        819        819          0          0          0          0
 10    8640524      22006          0          0          0          0       1261       1261      19484          0          0          0
 11     299711        105          0          0          0          0         40         40         25          0          0         18
 12    3513079       1030          0          0          0          0        515        515          0          0          0          0
 13    2905492        870          0          0          0          0        435        435          0          0          0          0
 14     391698        162          0          0          0          0         48         48         66          0          0         72
 15    9289446       2664          0          0          0          0       1321       1321         22          0          0         12
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.203s (  993.692 blocks/s,  62.106 Mb/s)
Attachment 30: Screenshot_from_2024-04-22_12-31-14.png
Screenshot_from_2024-04-22_12-31-14.png
Attachment 31: Screenshot_from_2024-04-22_12-31-41.png
Screenshot_from_2024-04-22_12-31-41.png
Attachment 32: Screenshot_from_2024-04-22_12-32-08.png
Screenshot_from_2024-04-22_12-32-08.png
Attachment 33: Screenshot_from_2024-04-22_12-32-45.png
Screenshot_from_2024-04-22_12-32-45.png
Attachment 34: Screenshot_from_2024-04-22_13-48-06.png
Screenshot_from_2024-04-22_13-48-06.png
Attachment 35: Screenshot_from_2024-04-22_14-07-29.png
Screenshot_from_2024-04-22_14-07-29.png
Attachment 36: Screenshot_from_2024-04-22_14-09-23.png
Screenshot_from_2024-04-22_14-09-23.png
Attachment 37: Screenshot_from_2024-04-22_14-12-50.png
Screenshot_from_2024-04-22_14-12-50.png
Attachment 38: Screenshot_from_2024-04-22_14-13-05.png
Screenshot_from_2024-04-22_14-13-05.png
  584   Sun Apr 21 00:34:33 2024 TDSunday 21 April
01.35 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
 
      ADC data item stats OK - attachment 3

      Merger OK - attachment 4

04.05 DSSSD bias & leakage current OK - Grafana - attachment 5

      FEE64 temperatures OK - attachment 6
 
      ADC data item stats OK - attachment 7

      Merger OK - attachment 8

08.20 DSSSD bias & leakage current OK - Grafana - attachment 9

      FEE64 temperatures OK - attachment 10
 
      ADC data item stats - attachment 11
       increased rates observed esp. DSSSD#2 n+n

      Merger etc - attachment 12
      rate increased x2 cf. overnight

      WR timestamp - attachment 13
       aida03 out sequence

09.25 MBS data relay restarted

      Merger data links - attachment 14
       aida03 rate zero

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16

      per FEE64 Rate spectra - attachment 17
      increased rate for DSSSD#2 n+n FEE64s in particular

      MIDASsort online analysis - attachments 18-19
      per DSSSD e_front versus e_back for LEC (20keV/channel) and HEC (20MeV/channel)
      observe light ions (LEC) c. 2MeV
      observe low rate of high energy ions (HEC) c. 600MeV
      similar energies in both DSSSDs for LEC and HEC implying both types of event have range >> 2mm Si


10.09 DAQ STOP
      aida03 does not respond - reboot aida03

      DAQ RESET/SETUP etc     

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Data transfer disabled
      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO

      WR timestamp OK - attachments 20-21

      ADC data item stats OK - attachment 22
       rates status quo ante
  
      Merger etc OK - attachment 23


14.58 DSSSD bias & leakage current OK - Grafana - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats OK - attachment 26

      Merger etc OK - attachment 27

18.13 DSSSD bias & leakage current OK - Grafana - attachment 28

      FEE64 temperatures OK - attachment 29

      ADC data item stats OK - attachment 30

      Merger etc OK - attachment 31


20.30 Primary c, few kHz 170Er beam to S4 - upstream s/steel flange removed
      Start of stopping range calibration in the implantation stack - currently ion appear to be stopping in DSSSD#1 (upstream)


      Decay and implant events as function of time showing spill structure - attachment 32

      per DSSSD e_front versus e_back (HEC, 20MeV/channel) - attachment 33

      per DSSSD x versus y hit pattern (HEC) - attachment 34
       mapping to be configured but clear ions are mostly implanted in centre of DSSSD#1 

      2D colour/contour levels - attachment 35

      per DSSSD m_front versus m_back (HEC) - attachment 36

      per DSSSD per pixel HEC-HEC time (4.096us/channel) - attachment 37

23.44 aida02 has stopped running
      DAQ STOP
      aida02 rebooted

      Data transfer disabled

      DAQ RESET/SETUP 

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO







        


        
Attachment 1: Screenshot_from_2024-04-21_01-31-59.png
Screenshot_from_2024-04-21_01-31-59.png
Attachment 2: Screenshot_from_2024-04-21_01-32-50.png
Screenshot_from_2024-04-21_01-32-50.png
Attachment 3: Screenshot_from_2024-04-21_01-33-28.png
Screenshot_from_2024-04-21_01-33-28.png
Attachment 4: Screenshot_from_2024-04-21_01-34-10.png
Screenshot_from_2024-04-21_01-34-10.png
Attachment 5: Screenshot_from_2024-04-21_04-05-57.png
Screenshot_from_2024-04-21_04-05-57.png
Attachment 6: Screenshot_from_2024-04-21_04-06-58.png
Screenshot_from_2024-04-21_04-06-58.png
Attachment 7: Screenshot_from_2024-04-21_04-07-36.png
Screenshot_from_2024-04-21_04-07-36.png
Attachment 8: Screenshot_from_2024-04-21_04-08-08.png
Screenshot_from_2024-04-21_04-08-08.png
Attachment 9: Screenshot_from_2024-04-21_08-18-51.png
Screenshot_from_2024-04-21_08-18-51.png
Attachment 10: Screenshot_from_2024-04-21_08-19-50.png
Screenshot_from_2024-04-21_08-19-50.png
Attachment 11: Screenshot_from_2024-04-21_08-21-20.png
Screenshot_from_2024-04-21_08-21-20.png
Attachment 12: Screenshot_from_2024-04-21_08-22-42.png
Screenshot_from_2024-04-21_08-22-42.png
Attachment 13: Screenshot_from_2024-04-21_08-28-22.png
Screenshot_from_2024-04-21_08-28-22.png
Attachment 14: Screenshot_from_2024-04-21_09-27-50.png
Screenshot_from_2024-04-21_09-27-50.png
Attachment 15: Screenshot_from_2024-04-21_09-29-07.png
Screenshot_from_2024-04-21_09-29-07.png
Attachment 16: Screenshot_from_2024-04-21_09-30-08.png
Screenshot_from_2024-04-21_09-30-08.png
Attachment 17: Screenshot_from_2024-04-21_09-31-12.png
Screenshot_from_2024-04-21_09-31-12.png
Attachment 18: Screenshot_from_2024-04-21_09-37-11.png
Screenshot_from_2024-04-21_09-37-11.png
Attachment 19: Screenshot_from_2024-04-21_09-38-08.png
Screenshot_from_2024-04-21_09-38-08.png
Attachment 20: Screenshot_from_2024-04-21_10-18-25.png
Screenshot_from_2024-04-21_10-18-25.png
Attachment 21: Screenshot_from_2024-04-21_10-32-41.png
Screenshot_from_2024-04-21_10-32-41.png
Attachment 22: Screenshot_from_2024-04-21_10-33-08.png
Screenshot_from_2024-04-21_10-33-08.png
Attachment 23: Screenshot_from_2024-04-21_10-33-33.png
Screenshot_from_2024-04-21_10-33-33.png
Attachment 24: Screenshot_from_2024-04-21_14-58-07.png
Screenshot_from_2024-04-21_14-58-07.png
Attachment 25: Screenshot_from_2024-04-21_14-58-31.png
Screenshot_from_2024-04-21_14-58-31.png
Attachment 26: Screenshot_from_2024-04-21_14-58-48.png
Screenshot_from_2024-04-21_14-58-48.png
Attachment 27: Screenshot_from_2024-04-21_14-59-00.png
Screenshot_from_2024-04-21_14-59-00.png
Attachment 28: Screenshot_from_2024-04-21_18-13-44.png
Screenshot_from_2024-04-21_18-13-44.png
Attachment 29: Screenshot_from_2024-04-21_18-14-06.png
Screenshot_from_2024-04-21_18-14-06.png
Attachment 30: Screenshot_from_2024-04-21_18-14-24.png
Screenshot_from_2024-04-21_18-14-24.png
Attachment 31: Screenshot_from_2024-04-21_18-14-40.png
Screenshot_from_2024-04-21_18-14-40.png
Attachment 32: Screenshot_from_2024-04-21_20-54-50.png
Screenshot_from_2024-04-21_20-54-50.png
Attachment 33: Screenshot_from_2024-04-21_20-54-18.png
Screenshot_from_2024-04-21_20-54-18.png
Attachment 34: Screenshot_from_2024-04-21_20-54-03.png
Screenshot_from_2024-04-21_20-54-03.png
Attachment 35: Screenshot_from_2024-04-21_20-53-41.png
Screenshot_from_2024-04-21_20-53-41.png
Attachment 36: Screenshot_from_2024-04-21_20-53-31.png
Screenshot_from_2024-04-21_20-53-31.png
Attachment 37: Screenshot_from_2024-04-21_21-02-02.png
Screenshot_from_2024-04-21_21-02-02.png
  583   Sat Apr 20 17:52:27 2024 TD, NHSaturday 20 April contd.
Note 40kBq 60Co source placed on bottom surface of snout centred on bPlas detectors and DSSSDs

18.50 Primary beam 750MeV/u Er to S4
      Primary beam stopped by flange (5cm s/steel in position upstream of implantation stack)

      Observe c. 2MeV dE peak in AIDA DSSSD#1 and DSSSD#2 using MIDASsort online analysis - attachment 1



21.50 Primary beam 1000MeV/u Er to S4 - attachment 2

      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections

10.50 Overnight stats - attachment 3
      Primary beam ?
      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections
      note multiple dE peaks observed in both DSSSDs
Attachment 1: Screenshot_from_2024-04-20_18-51-53.png
Screenshot_from_2024-04-20_18-51-53.png
Attachment 2: Screenshot_from_2024-04-20_21-52-08.png
Screenshot_from_2024-04-20_21-52-08.png
Attachment 3: Screenshot_from_2024-04-21_10-52-19.png
Screenshot_from_2024-04-21_10-52-19.png
  582   Sat Apr 20 11:40:38 2024 TDOffline analysis R12 & R13 ( pulser walkthrough )
S100 ADC offsets using pulser walkthrough data from data files R12 & R13

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )




per p+n FEE64 ADC spectra (aida01-aida16 *not* aida02, aida04, aida06 & aida08) - attachments 1-12

Note common x & y-scales - peak height proxy for peak width

aida09 1.8.L (s520) pulser peak width 7 ch FWHM (~40keV FWHM)




per n+n FEE64 ADC spectra (aida02, aida04, aida06 & aida08) - attachments 13-16

Note common x & y-scales - peak height proxy for peak width

aida02 1.8.L (s72) pulser peak width 16 ch FWHM (~88keV FWHM)




ADC offset analysis

/home/npg/td/Calibration/offsets /home/npg/td/S100/R12-R13 50 1024 20 5


First pass sort variables ( ADC offsets, FEE64 configuration, LEC & HEC equal energy cuts ) - attachment 17
Attachment 1: Screenshot_from_2024-04-20_12-36-14.png
Screenshot_from_2024-04-20_12-36-14.png
Attachment 2: Screenshot_from_2024-04-20_12-36-35.png
Screenshot_from_2024-04-20_12-36-35.png
Attachment 3: Screenshot_from_2024-04-20_12-36-55.png
Screenshot_from_2024-04-20_12-36-55.png
Attachment 4: Screenshot_from_2024-04-20_12-37-21.png
Screenshot_from_2024-04-20_12-37-21.png
Attachment 5: Screenshot_from_2024-04-20_12-37-55.png
Screenshot_from_2024-04-20_12-37-55.png
Attachment 6: Screenshot_from_2024-04-20_12-38-22.png
Screenshot_from_2024-04-20_12-38-22.png
Attachment 7: Screenshot_from_2024-04-20_12-38-47.png
Screenshot_from_2024-04-20_12-38-47.png
Attachment 8: Screenshot_from_2024-04-20_12-39-05.png
Screenshot_from_2024-04-20_12-39-05.png
Attachment 9: Screenshot_from_2024-04-20_12-39-21.png
Screenshot_from_2024-04-20_12-39-21.png
Attachment 10: Screenshot_from_2024-04-20_12-39-34.png
Screenshot_from_2024-04-20_12-39-34.png
Attachment 11: Screenshot_from_2024-04-20_12-39-52.png
Screenshot_from_2024-04-20_12-39-52.png
Attachment 12: Screenshot_from_2024-04-20_12-40-17.png
Screenshot_from_2024-04-20_12-40-17.png
Attachment 13: Screenshot_from_2024-04-20_12-51-16.png
Screenshot_from_2024-04-20_12-51-16.png
Attachment 14: Screenshot_from_2024-04-20_12-51-38.png
Screenshot_from_2024-04-20_12-51-38.png
Attachment 15: Screenshot_from_2024-04-20_12-52-00.png
Screenshot_from_2024-04-20_12-52-00.png
Attachment 16: Screenshot_from_2024-04-20_12-52-53.png
Screenshot_from_2024-04-20_12-52-53.png
Attachment 17: variables.dat
  $variables
  offset(   0) =    34.20
  offset(   1) =    27.50
  offset(   2) =   -18.51
  offset(   3) =    13.56
  offset(   4) =    -6.12
  offset(   5) =   -14.51
  offset(   6) =    11.86
  offset(   7) =    -0.19
  offset(   8) =     2.52
  offset(   9) =    24.94
  offset(  10) =     7.54
  offset(  11) =   -20.01
  offset(  12) =    -0.93
  offset(  13) =    26.51
  offset(  14) =   -12.75
  offset(  15) =    -4.72
  offset(  16) =     5.48
  offset(  17) =    -8.97
  offset(  18) =     4.25
  offset(  19) =    -3.19
  offset(  20) =   -40.78
  offset(  21) =    -0.95
  offset(  22) =   -14.62
  offset(  23) =   -28.18
  offset(  24) =   -12.96
  offset(  25) =    14.90
  offset(  26) =   -32.39
  offset(  27) =     0.66
  offset(  28) =     6.68
  offset(  29) = -9999.99
  offset(  30) =    15.03
  offset(  31) =    -2.98
  offset(  32) =    16.22
  offset(  33) =    27.02
  offset(  34) = -9999.99
  offset(  35) =    17.09
  offset(  36) =    17.72
  offset(  37) =    16.29
  offset(  38) =    23.46
  offset(  39) =    27.55
  offset(  40) =     9.26
  offset(  41) =    34.03
  offset(  42) =     2.46
  offset(  43) =     8.62
  offset(  44) =    29.61
  offset(  45) =   -14.47
  offset(  46) =    42.23
  offset(  47) =   -27.52
  offset(  48) =     1.06
  offset(  49) =     4.03
  offset(  50) =    22.04
  offset(  51) =   -10.44
  offset(  52) =    38.54
  offset(  53) =     8.66
  offset(  54) =    20.65
  offset(  55) =     0.35
  offset(  56) =     5.19
  offset(  57) =    19.55
  offset(  58) =    -5.57
  offset(  59) =    -7.26
  offset(  60) =   -21.96
  offset(  61) =     5.89
  offset(  62) =    36.14
  offset(  63) =     3.05
  offset(  64) =   -15.80
  offset(  65) =    25.02
  offset(  66) =   -15.94
  offset(  67) =     6.19
  offset(  68) =   -19.87
  offset(  69) =     0.37
  offset(  70) =    -9.68
  offset(  71) =     9.68
  offset(  72) =   -11.13
  offset(  73) =    -6.20
  offset(  74) =   -25.35
  offset(  75) =   -25.57
  offset(  76) =   -19.75
  offset(  77) =     8.60
  offset(  78) =   -22.55
  offset(  79) =   -10.56
  offset(  80) =     5.90
  offset(  81) =    -1.29
  offset(  82) =    -1.52
  offset(  83) = -9999.99
  offset(  84) =    24.56
  offset(  85) =   -10.73
  offset(  86) =    11.96
  offset(  87) =   -35.20
  offset(  88) =    16.67
  offset(  89) =    10.31
  offset(  90) =   -13.86
  offset(  91) =   -33.64
  offset(  92) =     2.96
  offset(  93) =    -2.41
  offset(  94) =     1.80
  offset(  95) =    26.72
  offset(  96) =   -18.06
  offset(  97) =    -0.82
  offset(  98) =   -19.61
  offset(  99) = -9999.99
  offset( 100) =   -11.22
  offset( 101) =     9.13
  offset( 102) =   -29.56
  offset( 103) =    -1.95
  offset( 104) =   -14.24
  offset( 105) =   -21.87
  offset( 106) =    -7.47
  offset( 107) =   -14.22
  offset( 108) =    -6.72
  offset( 109) =    -4.35
  offset( 110) =   -12.98
  offset( 111) =   -11.42
  offset( 112) =     5.39
  offset( 113) =    16.43
  offset( 114) =   -10.19
  offset( 115) =   -13.21
  offset( 116) =    19.82
  offset( 117) =    -4.45
  offset( 118) =    21.88
  offset( 119) =    13.84
  offset( 120) =    10.16
  offset( 121) = -9999.99
  offset( 122) = -9999.99
  offset( 123) = -9999.99
  offset( 124) =    13.42
  offset( 125) =     4.74
  offset( 126) =    10.81
  offset( 127) =    16.15
  offset( 128) =    -4.37
  offset( 129) =    21.60
  offset( 130) =     8.61
  offset( 131) =    -2.95
  offset( 132) =   -10.00
  offset( 133) =     2.40
  offset( 134) =    -9.40
  offset( 135) =   -24.68
  offset( 136) =     4.46
  offset( 137) =    -0.20
  offset( 138) =    20.63
  offset( 139) =   -25.14
  offset( 140) =    -7.48
  offset( 141) =    13.71
  offset( 142) =    -4.11
  offset( 143) =    -9.17
  offset( 144) =    15.36
  offset( 145) =    11.07
  offset( 146) =   -13.21
  offset( 147) =    -8.91
  offset( 148) =     4.75
  offset( 149) =   -10.47
  offset( 150) =    18.10
  offset( 151) =     0.62
  offset( 152) =    -2.18
  offset( 153) =     5.79
  offset( 154) =     2.73
  offset( 155) =    17.67
  offset( 156) =    15.58
  offset( 157) =   -34.45
  offset( 158) =    -1.55
  offset( 159) =    12.93
  offset( 160) =     4.16
  offset( 161) =   -15.55
  offset( 162) =    17.51
  offset( 163) =    -5.30
  offset( 164) =     8.65
  offset( 165) =   -13.48
  offset( 166) =   -10.14
  offset( 167) =    -3.75
  offset( 168) =    12.28
  offset( 169) =   -23.31
  offset( 170) =   -11.44
  offset( 171) =     5.06
  offset( 172) =   -15.18
  offset( 173) =    17.88
  offset( 174) =    12.26
  offset( 175) =   -13.64
  offset( 176) =    -9.33
  offset( 177) =    21.57
  offset( 178) =    28.82
  offset( 179) =    27.67
  offset( 180) =    62.22
  offset( 181) =    -2.89
  offset( 182) =    17.11
  offset( 183) =    11.80
  offset( 184) =    17.26
  offset( 185) =    15.52
  offset( 186) =    -8.89
  offset( 187) =    15.53
  offset( 188) =   -11.80
  offset( 189) =     0.85
  offset( 190) =    11.21
  offset( 191) =     7.76
  offset( 192) =     5.12
  offset( 193) =    26.29
  offset( 194) = -9999.99
  offset( 195) =     7.45
  offset( 196) = -9999.99
  offset( 197) =     2.31
  offset( 198) =    -1.94
  offset( 199) = -9999.99
  offset( 200) =    -0.41
  offset( 201) =   -23.18
  offset( 202) =    -1.82
  offset( 203) =    17.39
  offset( 204) =   -10.40
  offset( 205) =   -13.42
  offset( 206) =     8.22
  offset( 207) =     2.93
  offset( 208) =   -56.48
  offset( 209) =   -10.42
  offset( 210) =    50.72
  offset( 211) =     0.45
  offset( 212) =    27.74
  offset( 213) =   -16.62
  offset( 214) =    15.39
  offset( 215) =   -13.44
  offset( 216) =    34.58
  offset( 217) =    15.20
  offset( 218) =    -3.05
  offset( 219) =    -0.28
  offset( 220) =   -40.52
  offset( 221) =   -31.47
  offset( 222) =    20.29
  offset( 223) =   -13.69
  offset( 224) =    -1.11
  offset( 225) =    33.58
  offset( 226) =   -16.57
  offset( 227) =    -4.95
  offset( 228) =   -11.93
  offset( 229) =     7.06
  offset( 230) =    -2.90
  offset( 231) =   -38.90
  offset( 232) =    -8.84
  offset( 233) =     1.20
  offset( 234) =     3.21
  offset( 235) =    17.26
  offset( 236) =     4.76
  offset( 237) =    13.15
  offset( 238) =    13.66
  offset( 239) =    -2.42
  offset( 240) =    -1.46
  offset( 241) =     0.03
  offset( 242) =    20.72
  offset( 243) =    -0.74
  offset( 244) =    -2.03
  offset( 245) =   -36.41
  offset( 246) =     5.31
  offset( 247) =     3.45
  offset( 248) =    -9.35
  offset( 249) =     8.61
  offset( 250) =     6.18
  offset( 251) =   -72.01
  offset( 252) =    11.53
  offset( 253) =   -13.74
  offset( 254) =    -2.41
  offset( 255) =   -16.47
  offset( 256) =     6.19
  offset( 257) =    13.35
  offset( 258) =    10.44
  offset( 259) =    16.45
  offset( 260) =    13.62
  offset( 261) =    36.59
  offset( 262) =    24.89
  offset( 263) =    11.04
  offset( 264) =    12.12
  offset( 265) =    -1.04
  offset( 266) =    20.55
  offset( 267) =   -19.88
  offset( 268) =    10.57
  offset( 269) =    31.05
  offset( 270) =    23.09
  offset( 271) =   -17.38
  offset( 272) =   -18.40
  offset( 273) =    16.00
  offset( 274) =    -5.59
  offset( 275) =    -3.98
  offset( 276) =     0.70
  offset( 277) =     4.32
  offset( 278) =   -13.61
  offset( 279) =    26.62
  offset( 280) =    17.81
  offset( 281) =     7.10
  offset( 282) =   -30.97
  offset( 283) =    13.50
  offset( 284) =    88.96
  offset( 285) =    -0.03
  offset( 286) =     8.64
  offset( 287) =     1.86
  offset( 288) =     4.27
  offset( 289) =    19.21
  offset( 290) =   -33.21
  offset( 291) =   -28.54
  offset( 292) =     7.98
  offset( 293) =    -8.69
  offset( 294) =     1.27
  offset( 295) =    20.08
  offset( 296) =    15.92
  offset( 297) =    -4.67
  offset( 298) =    30.68
... 1771 more lines ...
  581   Sat Apr 20 03:20:23 2024 TDSaturday 20 April
Note 60Co source (current activity c. 43kBq) was installed yesterday evening - assume it is positioned on the bottom side of the snout as per the other sources used this week/

04.17 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 3
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5


11.24 DSSSD bias & leakage current OK - Grafana - attachment 6

      FEE64 temperatures OK - attachment 7
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 8
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 9

      Merger etc - attachment 10


17.50 DSSSD bias & leakage current OK - Grafana - attachment 11

      FEE64 temperatures OK - attachment 12
       aida02 ASIC temp known to be u/s

      ALl syhstem wide checks OK *except* WR/FPGA errors - attachments 13-14

      WR timestamps OK - attachment 15

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18

      Merger, TapeServer, Data Links - attachments 19-20
       no data storage

      ADC data item stats - attachment 21
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs


18.01 Grafana alerting shows detail of DSSSD leakage current and S4 temperature variations over time - attachment  22
       DEGAS LN2 fill cycle every 6h 10.00 - 16.00 - 22.00 ... etc

https://despec-vm-01.gsi.de/grafana/d/tHRVJYfGz/aida-alerting?orgId=1
Attachment 1: Screenshot_from_2024-04-20_04-17-19.png
Screenshot_from_2024-04-20_04-17-19.png
Attachment 2: Screenshot_from_2024-04-20_04-17-52.png
Screenshot_from_2024-04-20_04-17-52.png
Attachment 3: Screenshot_from_2024-04-20_04-18-26.png
Screenshot_from_2024-04-20_04-18-26.png
Attachment 4: Screenshot_from_2024-04-20_04-19-13.png
Screenshot_from_2024-04-20_04-19-13.png
Attachment 5: Screenshot_from_2024-04-20_04-19-58.png
Screenshot_from_2024-04-20_04-19-58.png
Attachment 6: Screenshot_from_2024-04-20_11-25-02.png
Screenshot_from_2024-04-20_11-25-02.png
Attachment 7: Screenshot_from_2024-04-20_11-25-54.png
Screenshot_from_2024-04-20_11-25-54.png
Attachment 8: Screenshot_from_2024-04-20_11-26-43.png
Screenshot_from_2024-04-20_11-26-43.png
Attachment 9: Screenshot_from_2024-04-20_11-27-20.png
Screenshot_from_2024-04-20_11-27-20.png
Attachment 10: Screenshot_from_2024-04-20_11-28-05.png
Screenshot_from_2024-04-20_11-28-05.png
Attachment 11: Screenshot_from_2024-04-20_17-50-12.png
Screenshot_from_2024-04-20_17-50-12.png
Attachment 12: Screenshot_from_2024-04-20_17-50-38.png
Screenshot_from_2024-04-20_17-50-38.png
Attachment 13: Screenshot_from_2024-04-20_17-51-20.png
Screenshot_from_2024-04-20_17-51-20.png
Attachment 14: Screenshot_from_2024-04-20_17-51-03.png
Screenshot_from_2024-04-20_17-51-03.png
Attachment 15: Screenshot_from_2024-04-20_17-54-41.png
Screenshot_from_2024-04-20_17-54-41.png
Attachment 16: Screenshot_from_2024-04-20_17-51-58.png
Screenshot_from_2024-04-20_17-51-58.png
Attachment 17: Screenshot_from_2024-04-20_17-52-58.png
Screenshot_from_2024-04-20_17-52-58.png
Attachment 18: Screenshot_from_2024-04-20_17-52-26.png
Screenshot_from_2024-04-20_17-52-26.png
Attachment 19: Screenshot_from_2024-04-20_17-53-32.png
Screenshot_from_2024-04-20_17-53-32.png
Attachment 20: Screenshot_from_2024-04-20_17-53-24.png
Screenshot_from_2024-04-20_17-53-24.png
Attachment 21: Screenshot_from_2024-04-20_17-53-15.png
Screenshot_from_2024-04-20_17-53-15.png
Attachment 22: Screenshot_from_2024-04-20_17-51-37.png
Screenshot_from_2024-04-20_17-51-37.png
Attachment 23: Screenshot_from_2024-04-20_18-05-01.png
Screenshot_from_2024-04-20_18-05-01.png
  580   Fri Apr 19 13:52:34 2024 JB, CC, TDFriday 19 April contd.
12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       
Attachment 1: Screenshot_from_2024-04-19_14-56-51.png
Screenshot_from_2024-04-19_14-56-51.png
Attachment 2: Screenshot_from_2024-04-19_14-57-04.png
Screenshot_from_2024-04-19_14-57-04.png
Attachment 3: Screenshot_from_2024-04-19_14-58-27.png
Screenshot_from_2024-04-19_14-58-27.png
Attachment 4: Screenshot_from_2024-04-19_14-58-23.png
Screenshot_from_2024-04-19_14-58-23.png
Attachment 5: Screenshot_from_2024-04-19_14-59-02.png
Screenshot_from_2024-04-19_14-59-02.png
Attachment 6: Screenshot_from_2024-04-19_15-01-31.png
Screenshot_from_2024-04-19_15-01-31.png
Attachment 7: Screenshot_from_2024-04-19_15-01-17.png
Screenshot_from_2024-04-19_15-01-17.png
Attachment 8: Screenshot_from_2024-04-19_15-01-02.png
Screenshot_from_2024-04-19_15-01-02.png
Attachment 9: Screenshot_from_2024-04-19_15-00-39.png
Screenshot_from_2024-04-19_15-00-39.png
Attachment 10: Screenshot_from_2024-04-19_14-59-51.png
Screenshot_from_2024-04-19_14-59-51.png
Attachment 11: Screenshot_from_2024-04-19_15-02-09.png
Screenshot_from_2024-04-19_15-02-09.png
Attachment 12: Screenshot_from_2024-04-19_15-04-36.png
Screenshot_from_2024-04-19_15-04-36.png
Attachment 13: Screenshot_from_2024-04-19_15-03-48.png
Screenshot_from_2024-04-19_15-03-48.png
Attachment 14: Screenshot_from_2024-04-19_16-56-36.png
Screenshot_from_2024-04-19_16-56-36.png
Attachment 15: Screenshot_from_2024-04-19_18-45-05.png
Screenshot_from_2024-04-19_18-45-05.png
Attachment 16: Screenshot_from_2024-04-19_22-47-43.png
Screenshot_from_2024-04-19_22-47-43.png
Attachment 17: Screenshot_from_2024-04-19_22-48-52.png
Screenshot_from_2024-04-19_22-48-52.png
Attachment 18: Screenshot_from_2024-04-19_22-49-31.png
Screenshot_from_2024-04-19_22-49-31.png
Attachment 19: Screenshot_from_2024-04-19_22-50-15.png
Screenshot_from_2024-04-19_22-50-15.png
Attachment 20: Screenshot_from_2024-04-19_22-51-23.png
Screenshot_from_2024-04-19_22-51-23.png
Attachment 21: Screenshot_from_2024-04-19_22-52-39.png
Screenshot_from_2024-04-19_22-52-39.png
Attachment 22: Screenshot_from_2024-04-19_22-52-14.png
Screenshot_from_2024-04-19_22-52-14.png
Attachment 23: Screenshot_from_2024-04-19_22-51-55.png
Screenshot_from_2024-04-19_22-51-55.png
  579   Fri Apr 19 02:35:14 2024 TDFriday 19 April
03.30


09.10



09.25 data file S100_alpha/R11
   
      BNC PB-5 pulser data

      amplitude 10.0V
      attenuation x10
      tau_d = 1ms
      polarity - ( initially + )
      frequency 25Hz

      ASIC settings2024Mar27-11.25.32
       slow comparator 0x64 all FEE64s


      aida02 pulser peak width 129 ch FWHM

      polsrity +

      aida09 pulser peak width 57 ch FWHM


09.40 switch to no storage

      p+n FEE64 slow comparators 0x64 -> 0xa
      n+n FEE64 slow comparators 0x64 -> 0xf

      radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.

      ADC data item stats - attachment 35
      9/16 < 20k, max 108k

      per FEE64 Rate spectra - attachment 36

10.44 152Eu source removed

      ADC data item stats - attachment 37
      9/16 < 20k, max 102k

      per FEE64 Rate spectra - attachment 38
Attachment 1: Screenshot_from_2024-04-19_03-31-31.png
Screenshot_from_2024-04-19_03-31-31.png
Attachment 2: Screenshot_from_2024-04-19_03-30-58.png
Screenshot_from_2024-04-19_03-30-58.png
Attachment 3: Screenshot_from_2024-04-19_03-30-45.png
Screenshot_from_2024-04-19_03-30-45.png
Attachment 4: Screenshot_from_2024-04-19_03-27-45.png
Screenshot_from_2024-04-19_03-27-45.png
Attachment 5: Screenshot_from_2024-04-19_03-26-26.png
Screenshot_from_2024-04-19_03-26-26.png
Attachment 6: Screenshot_from_2024-04-19_03-25-53.png
Screenshot_from_2024-04-19_03-25-53.png
Attachment 7: Screenshot_from_2024-04-19_03-24-58.png
Screenshot_from_2024-04-19_03-24-58.png
Attachment 8: Screenshot_from_2024-04-19_09-08-46.png
Screenshot_from_2024-04-19_09-08-46.png
Attachment 9: Screenshot_from_2024-04-19_09-09-19.png
Screenshot_from_2024-04-19_09-09-19.png
Attachment 10: Screenshot_from_2024-04-19_09-10-19.png
Screenshot_from_2024-04-19_09-10-19.png
Attachment 11: Screenshot_from_2024-04-19_09-10-10.png
Screenshot_from_2024-04-19_09-10-10.png
Attachment 12: Screenshot_from_2024-04-19_09-10-03.png
Screenshot_from_2024-04-19_09-10-03.png
Attachment 13: Screenshot_from_2024-04-19_09-10-51.png
Screenshot_from_2024-04-19_09-10-51.png
Attachment 14: Screenshot_from_2024-04-19_09-11-08.png
Screenshot_from_2024-04-19_09-11-08.png
Attachment 15: Screenshot_from_2024-04-19_09-10-59.png
Screenshot_from_2024-04-19_09-10-59.png
Attachment 16: Screenshot_from_2024-04-19_09-11-20.png
Screenshot_from_2024-04-19_09-11-20.png
Attachment 17: Screenshot_from_2024-04-19_09-11-46.png
Screenshot_from_2024-04-19_09-11-46.png
Attachment 18: Screenshot_from_2024-04-19_09-11-31.png
Screenshot_from_2024-04-19_09-11-31.png
Attachment 19: Screenshot_from_2024-04-19_09-17-48.png
Screenshot_from_2024-04-19_09-17-48.png
Attachment 20: Screenshot_from_2024-04-19_09-17-22.png
Screenshot_from_2024-04-19_09-17-22.png
Attachment 21: Screenshot_from_2024-04-19_09-14-14.png
Screenshot_from_2024-04-19_09-14-14.png
Attachment 22: Screenshot_from_2024-04-19_09-13-23.png
Screenshot_from_2024-04-19_09-13-23.png
Attachment 23: Screenshot_from_2024-04-19_09-12-08.png
Screenshot_from_2024-04-19_09-12-08.png
Attachment 24: Screenshot_from_2024-04-19_09-12-01.png
Screenshot_from_2024-04-19_09-12-01.png
Attachment 25: Screenshot_from_2024-04-19_09-11-54.png
Screenshot_from_2024-04-19_09-11-54.png
Attachment 26: Screenshot_from_2024-04-19_09-18-40.png
Screenshot_from_2024-04-19_09-18-40.png
Attachment 27: Screenshot_from_2024-04-19_09-18-22.png
Screenshot_from_2024-04-19_09-18-22.png
Attachment 28: Screenshot_from_2024-04-19_09-18-01.png
Screenshot_from_2024-04-19_09-18-01.png
Attachment 29: Screenshot_from_2024-04-19_09-29-33.png
Screenshot_from_2024-04-19_09-29-33.png
Attachment 30: Screenshot_from_2024-04-19_09-27-40.png
Screenshot_from_2024-04-19_09-27-40.png
Attachment 31: Screenshot_from_2024-04-19_09-42-01.png
Screenshot_from_2024-04-19_09-42-01.png
Attachment 32: Screenshot_from_2024-04-19_09-41-31.png
Screenshot_from_2024-04-19_09-41-31.png
Attachment 33: Screenshot_from_2024-04-19_09-40-02.png
Screenshot_from_2024-04-19_09-40-02.png
Attachment 34: Screenshot_from_2024-04-19_09-37-03.png
Screenshot_from_2024-04-19_09-37-03.png
Attachment 35: Screenshot_from_2024-04-19_09-46-35.png
Screenshot_from_2024-04-19_09-46-35.png
Attachment 36: Screenshot_from_2024-04-19_09-47-08.png
Screenshot_from_2024-04-19_09-47-08.png
Attachment 37: Screenshot_from_2024-04-19_10-44-19.png
Screenshot_from_2024-04-19_10-44-19.png
Attachment 38: Screenshot_from_2024-04-19_10-43-58.png
Screenshot_from_2024-04-19_10-43-58.png
  578   Thu Apr 18 13:59:55 2024 TD, JBWR timestamp OK, readout time not OK
WR timestamps appear to be OK but readout times fort aida03, aida04 and aida09 appear to be out of sequence. Attachment 1-3.

Reseated HDMI cable of aida09 and performed complete powercycle. Check the MACB HDMI port (appears to be loose) - still no change aida09 not collecting WR timestamps. Attachment 4-5.
Attachment 1: Screenshot_from_2024-04-18_14-59-38.png
Screenshot_from_2024-04-18_14-59-38.png
Attachment 2: Screenshot_from_2024-04-18_15-01-15.png
Screenshot_from_2024-04-18_15-01-15.png
Attachment 3: Screenshot_from_2024-04-18_15-01-10.png
Screenshot_from_2024-04-18_15-01-10.png
Attachment 4: Screenshot_from_2024-04-18_16-13-55.png
Screenshot_from_2024-04-18_16-13-55.png
Attachment 5: Screenshot_from_2024-04-18_16-13-48.png
Screenshot_from_2024-04-18_16-13-48.png
  577   Thu Apr 18 08:57:48 2024 JB, TDThursday 18 April
09.52 DAQ continues file S100_alpha/R5
       note merger not functioning correctly due to aida03, aida04 & aida09 out of WR sequence 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      WR timestamps - attachment 3
       aida03, aida04 & aida09 out of sequence

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 4-6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-10

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-12

      Merger, TapeServer & Link Data Rates - attachments 13-15

      Grafana - attachment 16
       DSSSD bias/leakage current trends OK


19.15 alpha background file S100_alpha/R10

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temps OK - attachment 18
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 19-21

      WR timestamps OK - attachment 22

      ADC data item stats - attachment 23

      Merger, TapeServer & Meger stats - attachments 24-26
       data rate c. 2.8Mb/s

      per FEE64 1.8.W spectra - 20us FSR - attachments 27-28

      correlation scaler stats & data link stats - attachments 29-30
       data rate dominated by MACB scaler inputs aida03 & aida04 ( c. 200kHz each )


22.16 DAQ continues Ok file S100_alpha/R10_35

      Merger, TapeServer & Meger stats - attachments 31-33
       data rate c. 2.8Mb/s

      ADC data item stats - attachment 34

      FEE64 temps OK - attachment 35
      *except* aida02 ASIC temp which is known to be u/s

      DSSSD bias & leakage current OK - attachment 36
Attachment 1: Screenshot_from_2024-04-18_09-52-14.png
Screenshot_from_2024-04-18_09-52-14.png
Attachment 2: Screenshot_from_2024-04-18_09-52-28.png
Screenshot_from_2024-04-18_09-52-28.png
Attachment 3: Screenshot_from_2024-04-18_09-53-30.png
Screenshot_from_2024-04-18_09-53-30.png
Attachment 4: Screenshot_from_2024-04-18_09-52-57.png
Screenshot_from_2024-04-18_09-52-57.png
Attachment 5: Screenshot_from_2024-04-18_09-52-45.png
Screenshot_from_2024-04-18_09-52-45.png
Attachment 6: Screenshot_from_2024-04-18_09-52-40.png
Screenshot_from_2024-04-18_09-52-40.png
Attachment 7: Screenshot_from_2024-04-18_09-53-48.png
Screenshot_from_2024-04-18_09-53-48.png
Attachment 8: Screenshot_from_2024-04-18_09-56-18.png
Screenshot_from_2024-04-18_09-56-18.png
Attachment 9: Screenshot_from_2024-04-18_09-55-07.png
Screenshot_from_2024-04-18_09-55-07.png
Attachment 10: Screenshot_from_2024-04-18_09-54-11.png
Screenshot_from_2024-04-18_09-54-11.png
Attachment 11: Screenshot_from_2024-04-18_09-57-30.png
Screenshot_from_2024-04-18_09-57-30.png
Attachment 12: Screenshot_from_2024-04-18_09-56-48.png
Screenshot_from_2024-04-18_09-56-48.png
Attachment 13: Screenshot_from_2024-04-18_09-59-16.png
Screenshot_from_2024-04-18_09-59-16.png
Attachment 14: Screenshot_from_2024-04-18_09-59-26.png
Screenshot_from_2024-04-18_09-59-26.png
Attachment 15: Screenshot_from_2024-04-18_09-59-38.png
Screenshot_from_2024-04-18_09-59-38.png
Attachment 16: Screenshot_from_2024-04-18_10-10-43.png
Screenshot_from_2024-04-18_10-10-43.png
Attachment 17: Screenshot_from_2024-04-18_19-12-32.png
Screenshot_from_2024-04-18_19-12-32.png
Attachment 18: Screenshot_from_2024-04-18_19-12-47.png
Screenshot_from_2024-04-18_19-12-47.png
Attachment 19: Screenshot_from_2024-04-18_19-13-11.png
Screenshot_from_2024-04-18_19-13-11.png
Attachment 20: Screenshot_from_2024-04-18_19-13-04.png
Screenshot_from_2024-04-18_19-13-04.png
Attachment 21: Screenshot_from_2024-04-18_19-12-57.png
Screenshot_from_2024-04-18_19-12-57.png
Attachment 22: Screenshot_from_2024-04-18_19-13-51.png
Screenshot_from_2024-04-18_19-13-51.png
Attachment 23: Screenshot_from_2024-04-18_19-14-02.png
Screenshot_from_2024-04-18_19-14-02.png
Attachment 24: Screenshot_from_2024-04-18_19-15-10.png
Screenshot_from_2024-04-18_19-15-10.png
Attachment 25: Screenshot_from_2024-04-18_19-14-32.png
Screenshot_from_2024-04-18_19-14-32.png
Attachment 26: Screenshot_from_2024-04-18_19-14-12.png
Screenshot_from_2024-04-18_19-14-12.png
Attachment 27: Screenshot_from_2024-04-18_19-12-14.png
Screenshot_from_2024-04-18_19-12-14.png
Attachment 28: Screenshot_from_2024-04-18_19-11-30.png
Screenshot_from_2024-04-18_19-11-30.png
Attachment 29: Screenshot_from_2024-04-18_19-31-12.png
Screenshot_from_2024-04-18_19-31-12.png
Attachment 30: Screenshot_from_2024-04-18_19-41-59.png
Screenshot_from_2024-04-18_19-41-59.png
Attachment 31: Screenshot_from_2024-04-18_22-16-42.png
Screenshot_from_2024-04-18_22-16-42.png
Attachment 32: Screenshot_from_2024-04-18_22-16-04.png
Screenshot_from_2024-04-18_22-16-04.png
Attachment 33: Screenshot_from_2024-04-18_22-15-41.png
Screenshot_from_2024-04-18_22-15-41.png
Attachment 34: Screenshot_from_2024-04-18_22-12-53.png
Screenshot_from_2024-04-18_22-12-53.png
Attachment 35: Screenshot_from_2024-04-18_22-12-13.png
Screenshot_from_2024-04-18_22-12-13.png
Attachment 36: Screenshot_from_2024-04-18_22-10-53.png
Screenshot_from_2024-04-18_22-10-53.png
  576   Tue Apr 16 23:37:58 2024 TDWednesday 17 April
00.40 Observed zero data fron aida03, aida04, aida09
      WR timestamps offset from other FEE64s

      rebooted aida03, aida04, aida09
      no change
      see attachments 14-17



18.25 rebooted aida03, aida04, aida09, aida15

      system wide checks OK *except* aida15 ADC not calibrated, FPGA decoder errors aida02, aida03
      ADC data item stats - attachment 28
      WR timestamps OK - attachment 29

      data file S100_alpha/R5 - alpha background

      Merger and TapeServer - attachments 30-31
       c. 1Mb/s to disk

21.05 FEE64 temps OK - attachment 44
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida15 ASDC not calibrated, WR timestam/FPGA errors - attachments 41-43

      WR timestamps- attachment 40
       aida03, aida04 and aida09 differ from other timestamps again

      Merger and TapeServer - attachments 38-39
       c. 0Mb/s to disk due to WR timestamp issues

      per 1.8.W spectra - 20us FSR - attachments 36-37

      per FEE64 Rate spectra - attachments 34-35

      per FEE64 1.8.L spectra - attachments 32-33
Attachment 1: Screenshot_from_2024-04-17_00-35-53.png
Screenshot_from_2024-04-17_00-35-53.png
Attachment 2: Screenshot_from_2024-04-17_00-34-46.png
Screenshot_from_2024-04-17_00-34-46.png
Attachment 3: Screenshot_from_2024-04-17_00-33-46.png
Screenshot_from_2024-04-17_00-33-46.png
Attachment 4: Screenshot_from_2024-04-17_00-32-53.png
Screenshot_from_2024-04-17_00-32-53.png
Attachment 5: Screenshot_from_2024-04-17_00-31-43.png
Screenshot_from_2024-04-17_00-31-43.png
Attachment 6: Screenshot_from_2024-04-17_00-30-24.png
Screenshot_from_2024-04-17_00-30-24.png
Attachment 7: Screenshot_from_2024-04-17_00-29-34.png
Screenshot_from_2024-04-17_00-29-34.png
Attachment 8: Screenshot_from_2024-04-17_00-28-43.png
Screenshot_from_2024-04-17_00-28-43.png
Attachment 9: Screenshot_from_2024-04-17_00-27-54.png
Screenshot_from_2024-04-17_00-27-54.png
Attachment 10: Screenshot_from_2024-04-17_00-27-31.png
Screenshot_from_2024-04-17_00-27-31.png
Attachment 11: Screenshot_from_2024-04-17_00-27-05.png
Screenshot_from_2024-04-17_00-27-05.png
Attachment 12: Screenshot_from_2024-04-17_00-26-32.png
Screenshot_from_2024-04-17_00-26-32.png
Attachment 13: Screenshot_from_2024-04-17_00-25-51.png
Screenshot_from_2024-04-17_00-25-51.png
Attachment 14: Screenshot_from_2024-04-17_01-05-47.png
Screenshot_from_2024-04-17_01-05-47.png
Attachment 15: Screenshot_from_2024-04-17_01-05-09.png
Screenshot_from_2024-04-17_01-05-09.png
Attachment 16: Screenshot_from_2024-04-17_01-01-20.png
Screenshot_from_2024-04-17_01-01-20.png
Attachment 17: Screenshot_from_2024-04-17_01-00-33.png
Screenshot_from_2024-04-17_01-00-33.png
Attachment 18: Screenshot_from_2024-04-17_18-00-08.png
Screenshot_from_2024-04-17_18-00-08.png
Attachment 19: Screenshot_from_2024-04-17_17-59-42.png
Screenshot_from_2024-04-17_17-59-42.png
Attachment 20: Screenshot_from_2024-04-17_17-58-30.png
Screenshot_from_2024-04-17_17-58-30.png
Attachment 21: Screenshot_from_2024-04-17_17-57-46.png
Screenshot_from_2024-04-17_17-57-46.png
Attachment 22: Screenshot_from_2024-04-17_17-57-12.png
Screenshot_from_2024-04-17_17-57-12.png
Attachment 23: Screenshot_from_2024-04-17_17-56-27.png
Screenshot_from_2024-04-17_17-56-27.png
Attachment 24: Screenshot_from_2024-04-17_17-55-54.png
Screenshot_from_2024-04-17_17-55-54.png
Attachment 25: Screenshot_from_2024-04-17_17-55-33.png
Screenshot_from_2024-04-17_17-55-33.png
Attachment 26: Screenshot_from_2024-04-17_17-54-35.png
Screenshot_from_2024-04-17_17-54-35.png
Attachment 27: Screenshot_from_2024-04-17_17-54-04.png
Screenshot_from_2024-04-17_17-54-04.png
Attachment 28: Screenshot_from_2024-04-17_18-23-26.png
Screenshot_from_2024-04-17_18-23-26.png
Attachment 29: Screenshot_from_2024-04-17_18-22-42.png
Screenshot_from_2024-04-17_18-22-42.png
Attachment 30: Screenshot_from_2024-04-17_18-25-37.png
Screenshot_from_2024-04-17_18-25-37.png
Attachment 31: Screenshot_from_2024-04-17_18-25-08.png
Screenshot_from_2024-04-17_18-25-08.png
Attachment 32: Screenshot_from_2024-04-17_21-06-56.png
Screenshot_from_2024-04-17_21-06-56.png
Attachment 33: Screenshot_from_2024-04-17_21-05-45.png
Screenshot_from_2024-04-17_21-05-45.png
Attachment 34: Screenshot_from_2024-04-17_21-04-52.png
Screenshot_from_2024-04-17_21-04-52.png
Attachment 35: Screenshot_from_2024-04-17_21-04-17.png
Screenshot_from_2024-04-17_21-04-17.png
Attachment 36: Screenshot_from_2024-04-17_21-03-15.png
Screenshot_from_2024-04-17_21-03-15.png
Attachment 37: Screenshot_from_2024-04-17_21-02-29.png
Screenshot_from_2024-04-17_21-02-29.png
Attachment 38: Screenshot_from_2024-04-17_21-01-57.png
Screenshot_from_2024-04-17_21-01-57.png
Attachment 39: Screenshot_from_2024-04-17_21-01-32.png
Screenshot_from_2024-04-17_21-01-32.png
Attachment 40: Screenshot_from_2024-04-17_21-00-34.png
Screenshot_from_2024-04-17_21-00-34.png
Attachment 41: Screenshot_from_2024-04-17_20-59-40.png
Screenshot_from_2024-04-17_20-59-40.png
Attachment 42: Screenshot_from_2024-04-17_20-59-16.png
Screenshot_from_2024-04-17_20-59-16.png
Attachment 43: Screenshot_from_2024-04-17_20-58-54.png
Screenshot_from_2024-04-17_20-58-54.png
Attachment 44: Screenshot_from_2024-04-17_20-58-02.png
Screenshot_from_2024-04-17_20-58-02.png
  575   Tue Apr 16 10:49:37 2024 TD, JB, CCTuesday 16 April
11.45 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temps OK - attachment 8
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder/FPGA status errors - attachments 6-7
       HDMI cabling disturbed during platform move?

      WR timestamps OK - attachment 5

      ADC data item stats - attachment 4
        10/16 < 20k, max 115k

      per FEE64 Rate spectra - attachments 3

      per 1.8.W spectra - 20us FSR - attachments 1-2

11.58 BB7 preamps unplugged. Noise conditions OK. Did not change considerably.
      11/16 < 20k (aida09 has 1 hot channel), max 114k Attachments 11-12.

12:34 Detector powered down to bring to safe state to check HDMI cables of aida02 and aida03.

15:06 Reseated aida02 and aida03. No more FPGA timestamp error. See attachment 13-14.

      Rates and histograms look good for pulser walk through. Test + and test - connected. Currently, Attachment 16-17.
      Temp OK Attachment 15. Bias OK Attachment 18.

      Data links aida02 and aida08 missing (?)

16:30 All slow comparator thresholds set to 0x64 for alpha background run during platform freeze.

      Initial rates and histograms attachment 19 - 20. aida04 noisy in ASIC 1.

      Temps OK. Attachment 21.
      MBS disc and tape setup in atttachment 22-23.
Attachment 1: Screenshot_from_2024-04-16_11-49-17.png
Screenshot_from_2024-04-16_11-49-17.png
Attachment 2: Screenshot_from_2024-04-16_11-48-28.png
Screenshot_from_2024-04-16_11-48-28.png
Attachment 3: Screenshot_from_2024-04-16_11-47-40.png
Screenshot_from_2024-04-16_11-47-40.png
Attachment 4: Screenshot_from_2024-04-16_11-46-48.png
Screenshot_from_2024-04-16_11-46-48.png
Attachment 5: Screenshot_from_2024-04-16_11-46-01.png
Screenshot_from_2024-04-16_11-46-01.png
Attachment 6: Screenshot_from_2024-04-16_11-45-15.png
Screenshot_from_2024-04-16_11-45-15.png
Attachment 7: Screenshot_from_2024-04-16_11-44-37.png
Screenshot_from_2024-04-16_11-44-37.png
Attachment 8: Screenshot_from_2024-04-16_11-44-06.png
Screenshot_from_2024-04-16_11-44-06.png
Attachment 9: Screenshot_from_2024-04-16_11-51-17.png
Screenshot_from_2024-04-16_11-51-17.png
Attachment 10: Screenshot_from_2024-04-16_11-50-22.png
Screenshot_from_2024-04-16_11-50-22.png
Attachment 11: Screenshot_from_2024-04-16_11-58-26.png
Screenshot_from_2024-04-16_11-58-26.png
Attachment 12: Screenshot_from_2024-04-16_11-58-08.png
Screenshot_from_2024-04-16_11-58-08.png
Attachment 13: Screenshot_from_2024-04-16_15-04-13.png
Screenshot_from_2024-04-16_15-04-13.png
Attachment 14: Screenshot_from_2024-04-16_15-04-07.png
Screenshot_from_2024-04-16_15-04-07.png
Attachment 15: Screenshot_from_2024-04-16_15-06-31.png
Screenshot_from_2024-04-16_15-06-31.png
Attachment 16: Screenshot_from_2024-04-16_15-06-12.png
Screenshot_from_2024-04-16_15-06-12.png
Attachment 17: Screenshot_from_2024-04-16_15-05-33.png
Screenshot_from_2024-04-16_15-05-33.png
Attachment 18: Screenshot_from_2024-04-16_15-08-04.png
Screenshot_from_2024-04-16_15-08-04.png
Attachment 19: Screenshot_from_2024-04-16_15-10-46.png
Screenshot_from_2024-04-16_15-10-46.png
Attachment 20: Screenshot_from_2024-04-16_17-17-35.png
Screenshot_from_2024-04-16_17-17-35.png
Attachment 21: Screenshot_from_2024-04-16_17-17-32.png
Screenshot_from_2024-04-16_17-17-32.png
Attachment 22: Screenshot_from_2024-04-16_17-14-40.png
Screenshot_from_2024-04-16_17-14-40.png
Attachment 23: Screenshot_from_2024-04-16_17-17-24.png
Screenshot_from_2024-04-16_17-17-24.png
Attachment 24: Screenshot_from_2024-04-16_17-17-11.png
Screenshot_from_2024-04-16_17-17-11.png
  574   Mon Apr 15 09:23:43 2024 CC, JB, HMA, TDNoise in AIDA when connecting BB7

9:40 started up AIDA. TEMP OK. Noise condition same as Saturday. Attachments 1-3.

9:56 BB7 premaps connected but not powered. Attahcment 4.

10:20 BB7 Preamps powered. Attachment 5.

Noise increased marginally, but still 10/16 <20kHz, max 130k.

 

 

15.15  per FEE64 Rate spectra - attachment 6

             ADC data item stats - attachment 7

           increased noise observed - 6/16 < 20k, max 300k 

 

17.10  per FEE64 1.8.W spectra - 20us FSR - attachments 8-9

            per FEE64 Rate spectra - attachment 10

             ADC data item stats - attachment 11

           increased noise observed - 7/16 < 20k, max 180k 

20.30  per FEE64 Rate spectra - attachment 12

             ADC data item stats - attachment 13

           status quo ante - 10/16 < 20k, max 130k 

 

          FEE64 power OFF

          DSSSD bias OFF

 

 

Attachment 1: Screenshot_from_2024-04-15_09-55-15.png
Screenshot_from_2024-04-15_09-55-15.png
Attachment 2: Screenshot_from_2024-04-15_09-54-58.png
Screenshot_from_2024-04-15_09-54-58.png
Attachment 3: Screenshot_from_2024-04-15_09-54-17.png
Screenshot_from_2024-04-15_09-54-17.png
Attachment 4: Screenshot_from_2024-04-15_10-13-20.png
Screenshot_from_2024-04-15_10-13-20.png
Attachment 5: Screenshot_from_2024-04-15_10-24-39.png
Screenshot_from_2024-04-15_10-24-39.png
Attachment 6: Screenshot_from_2024-04-15_15-24-34.png
Screenshot_from_2024-04-15_15-24-34.png
Attachment 7: Screenshot_from_2024-04-15_15-23-41.png
Screenshot_from_2024-04-15_15-23-41.png
Attachment 8: Screenshot_from_2024-04-15_17-16-19.png
Screenshot_from_2024-04-15_17-16-19.png
Attachment 9: Screenshot_from_2024-04-15_17-15-27.png
Screenshot_from_2024-04-15_17-15-27.png
Attachment 10: Screenshot_from_2024-04-15_17-14-32.png
Screenshot_from_2024-04-15_17-14-32.png
Attachment 11: Screenshot_from_2024-04-15_17-13-37.png
Screenshot_from_2024-04-15_17-13-37.png
Attachment 12: Screenshot_from_2024-04-15_20-29-18.png
Screenshot_from_2024-04-15_20-29-18.png
Attachment 13: Screenshot_from_2024-04-15_20-28-48.png
Screenshot_from_2024-04-15_20-28-48.png
  573   Sat Apr 13 14:04:44 2024 JB13 April Noise checks

15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).

          TEMP OK.

          Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.

          9/16 <20kHz, max 176k

15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.

          No change in noise observed.

          9/16 <20kHz, max 187k

15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.

          10/16 <20kHz, max 158k.

 

Current bPlast grounding scheme in attachments 8-9.

 

Attachment 1: Screenshot_from_2024-04-13_15-21-34.png
Screenshot_from_2024-04-13_15-21-34.png
Attachment 2: Screenshot_from_2024-04-13_15-20-28.png
Screenshot_from_2024-04-13_15-20-28.png
Attachment 3: Screenshot_from_2024-04-13_15-15-14.png
Screenshot_from_2024-04-13_15-15-14.png
Attachment 4: Screenshot_from_2024-04-13_15-40-59.png
Screenshot_from_2024-04-13_15-40-59.png
Attachment 5: Screenshot_from_2024-04-13_15-40-56.png
Screenshot_from_2024-04-13_15-40-56.png
Attachment 6: Screenshot_from_2024-04-13_15-48-53.png
Screenshot_from_2024-04-13_15-48-53.png
Attachment 7: Screenshot_from_2024-04-13_15-48-42.png
Screenshot_from_2024-04-13_15-48-42.png
Attachment 8: 20240413_155823.jpg
20240413_155823.jpg
Attachment 9: 20240413_160214.jpg
20240413_160214.jpg
  572   Thu Apr 11 22:04:50 2024 JB12.04.2024 AIDA-bPlast noise optimisation

15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.

           Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.

           Histograms okay, some FEE64s now have hot channels. Attachment 2.

           9/16 aidas < 20kHz - max rate 162k

 

18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.

 

TO-DO (kicking the can down the road):

- Some work to do on noise, but we might have to accept the situation as it currently is.

- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.

- Try different grounding configuration:

    - Grounding drain wires to frame. Observe AIDA conditions.

    - Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.

 

Attachment 1: Screenshot_from_2024-04-12_15-11-18.png
Screenshot_from_2024-04-12_15-11-18.png
Attachment 2: Screenshot_from_2024-04-12_15-22-54.png
Screenshot_from_2024-04-12_15-22-54.png
Attachment 3: Screenshot_from_2024-04-12_15-22-26.png
Screenshot_from_2024-04-12_15-22-26.png
Attachment 4: Screenshot_from_2024-04-12_18-07-35.png
Screenshot_from_2024-04-12_18-07-35.png
Attachment 5: Screenshot_from_2024-04-12_18-07-27.png
Screenshot_from_2024-04-12_18-07-27.png
  571   Thu Apr 11 08:27:21 2024 JB, TD11 April noise tests
TO-DO for 11.04.2024

- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.

- Recheck the downstream detector bias and ground scheme.

-

 

9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,

        TEMP OK - HV OK. See attachments 1-6.

 

10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.

10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.

 

11.00 DSSSD HV OFF

          FEE64 power OFF

          Check seating of all adaptor PCBs and drain/ground wires secured - generally OK

 

          Restart

          1.8.W spectra - 20us FSR - attachments 15-16

          per FEE64 Rate spectra - attachment 17

           10/16 < 20k, max 110k

          ADC data item stats - attachment 18

          Incremental improvement.

 

11.45 DSSSD HV OFF

          FEE64 power OFF

          test - daisy chain removed

 

          Restart

          1.8.W spectra - 20us FSR - attachments 20-21

          per FEE64 Rate spectra - attachment 19

           10/16 < 20k, max 210k

          ADC data item stats - attachment 22

          overall somewhat worse

 

12.20 DSSSD HV OFF

          FEE64 power OFF

          Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.

 

          Restart

          1.8.W spectra - 20us FSR - attachments 23-24

          per FEE64 Rate spectra - attachment 25

          ADC data item stats - attachment 26

           10/16 < 20k, max 110k

          Status quo ante

 

         aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31

           large transients observed for aida04 1*W - cable or ASIC fault?

 

         aida16 **W spectra - 200us FSR - attachments 32-35

           all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise

 

       

 

 

        

14.00 DSSSD HV OFF

          FEE64 power OFF

          Disconnect ribbon cables from aida04 adaptor PCB

 

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 37

          per FEE64 Rate spectra - attachment 36

          ADC data item stats - attachment 38

          Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.

 

 

14.40 DSSSD HV OFF

          FEE64 power OFF

          re-connect ribbon cables from aida04 adaptor PCB

          1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK

 

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41

          per FEE64 Rate spectra - attachment 42-43

          ADC data item stats - attachment 39

            10x < 20k, max 120k

 

 

           aida11 asic#4 1.8.W spectra - 200us FSR - attachment

             no large transients observed

 

           DSSSD #1 & #2 bias from -120V to -100V

            ADC data item stats - attachment 40

            9x < 20k, max 220k (aida08)

            per FEE64 Rate spectra - attachment 41

 

17.15 bPlas ON -

        Current bPlast ground configuration:

         bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at

         29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.

         It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.         

          AIDA noise very good

            ADC data item stats - attachment 42

            12x < 20k, max 83k (aida04)

            per FEE64 Rate spectra - attachment 43

           

 

 

To Do list

- separate 29V (low current) and 4V (high current) return paths/ground refs

- The mesytec PSU that powers the bPlast detector  is running to frame, it might be an idea to ground ribbon c

 

Attachment 1: Screenshot_from_2024-04-11_09-56-51.png
Screenshot_from_2024-04-11_09-56-51.png
Attachment 2: Screenshot_from_2024-04-11_09-55-50.png
Screenshot_from_2024-04-11_09-55-50.png
Attachment 3: Screenshot_from_2024-04-11_09-55-21.png
Screenshot_from_2024-04-11_09-55-21.png
Attachment 4: Screenshot_from_2024-04-11_09-55-10.png
Screenshot_from_2024-04-11_09-55-10.png
Attachment 5: Screenshot_from_2024-04-11_09-54-19.png
Screenshot_from_2024-04-11_09-54-19.png
Attachment 6: Screenshot_from_2024-04-11_09-53-48.png
Screenshot_from_2024-04-11_09-53-48.png
Attachment 7: Screenshot_from_2024-04-11_10-21-37.png
Screenshot_from_2024-04-11_10-21-37.png
Attachment 8: Screenshot_from_2024-04-11_10-21-08.png
Screenshot_from_2024-04-11_10-21-08.png
Attachment 9: Screenshot_from_2024-04-11_10-18-40.png
Screenshot_from_2024-04-11_10-18-40.png
Attachment 10: Screenshot_from_2024-04-11_10-18-38.png
Screenshot_from_2024-04-11_10-18-38.png
Attachment 11: Screenshot_from_2024-04-11_10-38-51.png
Screenshot_from_2024-04-11_10-38-51.png
Attachment 12: Screenshot_from_2024-04-11_10-37-16.png
Screenshot_from_2024-04-11_10-37-16.png
Attachment 13: Screenshot_from_2024-04-11_10-36-23.png
Screenshot_from_2024-04-11_10-36-23.png
Attachment 14: Screenshot_from_2024-04-11_10-36-13.png
Screenshot_from_2024-04-11_10-36-13.png
Attachment 15: Screenshot_from_2024-04-11_11-18-05.png
Screenshot_from_2024-04-11_11-18-05.png
Attachment 16: Screenshot_from_2024-04-11_11-17-16.png
Screenshot_from_2024-04-11_11-17-16.png
Attachment 17: Screenshot_from_2024-04-11_11-14-20.png
Screenshot_from_2024-04-11_11-14-20.png
Attachment 18: Screenshot_from_2024-04-11_11-14-08.png
Screenshot_from_2024-04-11_11-14-08.png
Attachment 19: Screenshot_from_2024-04-11_11-47-55.png
Screenshot_from_2024-04-11_11-47-55.png
Attachment 20: Screenshot_from_2024-04-11_11-46-52.png
Screenshot_from_2024-04-11_11-46-52.png
Attachment 21: Screenshot_from_2024-04-11_11-46-01.png
Screenshot_from_2024-04-11_11-46-01.png
Attachment 22: Screenshot_from_2024-04-11_11-45-32.png
Screenshot_from_2024-04-11_11-45-32.png
Attachment 23: Screenshot_from_2024-04-11_12-24-14.png
Screenshot_from_2024-04-11_12-24-14.png
Attachment 24: Screenshot_from_2024-04-11_12-23-39.png
Screenshot_from_2024-04-11_12-23-39.png
Attachment 25: Screenshot_from_2024-04-11_12-23-08.png
Screenshot_from_2024-04-11_12-23-08.png
Attachment 26: Screenshot_from_2024-04-11_12-22-27.png
Screenshot_from_2024-04-11_12-22-27.png
Attachment 27: Screenshot_from_2024-04-11_12-35-36.png
Screenshot_from_2024-04-11_12-35-36.png
Attachment 28: Screenshot_from_2024-04-11_12-35-07.png
Screenshot_from_2024-04-11_12-35-07.png
Attachment 29: Screenshot_from_2024-04-11_12-34-18.png
Screenshot_from_2024-04-11_12-34-18.png
Attachment 30: Screenshot_from_2024-04-11_12-33-13.png
Screenshot_from_2024-04-11_12-33-13.png
Attachment 31: Screenshot_from_2024-04-11_12-37-22.png
Screenshot_from_2024-04-11_12-37-22.png
Attachment 32: Screenshot_from_2024-04-11_12-44-59.png
Screenshot_from_2024-04-11_12-44-59.png
Attachment 33: Screenshot_from_2024-04-11_12-43-07.png
Screenshot_from_2024-04-11_12-43-07.png
Attachment 34: Screenshot_from_2024-04-11_12-42-19.png
Screenshot_from_2024-04-11_12-42-19.png
Attachment 35: Screenshot_from_2024-04-11_12-40-03.png
Screenshot_from_2024-04-11_12-40-03.png
Attachment 36: Screenshot_from_2024-04-11_14-04-59.png
Screenshot_from_2024-04-11_14-04-59.png
Attachment 37: Screenshot_from_2024-04-11_14-05-23.png
Screenshot_from_2024-04-11_14-05-23.png
Attachment 38: Screenshot_from_2024-04-11_14-19-10.png
Screenshot_from_2024-04-11_14-19-10.png
Attachment 39: Screenshot_from_2024-04-11_14-48-09.png
Screenshot_from_2024-04-11_14-48-09.png
Attachment 40: Screenshot_from_2024-04-11_14-48-29.png
Screenshot_from_2024-04-11_14-48-29.png
Attachment 41: Screenshot_from_2024-04-11_14-48-57.png
Screenshot_from_2024-04-11_14-48-57.png
Attachment 42: Screenshot_from_2024-04-11_14-49-37.png
Screenshot_from_2024-04-11_14-49-37.png
Attachment 43: Screenshot_from_2024-04-11_14-49-26.png
Screenshot_from_2024-04-11_14-49-26.png
Attachment 44: Screenshot_from_2024-04-11_15-11-42.png
Screenshot_from_2024-04-11_15-11-42.png
Attachment 45: Screenshot_from_2024-04-11_15-17-34.png
Screenshot_from_2024-04-11_15-17-34.png
Attachment 46: Screenshot_from_2024-04-11_15-17-10.png
Screenshot_from_2024-04-11_15-17-10.png
Attachment 47: Screenshot_from_2024-04-11_17-13-36.png
Screenshot_from_2024-04-11_17-13-36.png
Attachment 48: Screenshot_from_2024-04-11_17-13-40.png
Screenshot_from_2024-04-11_17-13-40.png
  570   Wed Apr 10 08:37:57 2024 JB, CC, TDWednesday 10 April
09.38 CC completed install of Bplast driver PCBs yesterday evening.
      All flat ribbon cables connected - all drain wires grounded
      PSU on but not enabled - return terminals grounded to PSU front panel ground
      SiPm bias off

      Water pressure and temperature OK
      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      ADC data item stats - attachments 5

      per FEE64 Rate spectra - attachments 6-7

      per 1.8.W spectra - 20us FSR - attachments 8-11

      ASIC settings 2024Mar27-11.25.32
       LEC slow comparator p+n FEE64s 0xa, n+n FEE64s 0xf

      BNC PB-5 pulser - attachment 12



10.45 CC returns
      bPlas ON

      ADC data item stats - attachments 13
       8x < 20k, max c. 310k

      per FEE64 Rate spectra - attachments 14

      per 1.8.W spectra - 20us FSR - attachments 15-16

      downstream DSSSD n+n and bottom left & right p+n FEE64s noisy


11.40 per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 55 ch FWHM ~38keV FWHM - no change cf. before installation of bPas *except* aida16


12.00 Photos of snout, bPlas driver PCBs, cabling, grounding and PSUs courtesy JB - attachments 18-27

      N.B outputs of PN300 PSU at base of AIDA support stand are *not* ground ref'd - attachment 23


12.20 Slow comparator -> 0x64
      Pulser OFF
      All histograms zero'd

13.17 JB: returned from lunch. Current status of AIDA modules given by attachment 29.

15.55 While bPlast thresholds were being set the noise increased substantially.

16.51 Replaced mezzanine of aida14, reinstalled and biasing detector. Resulting for noise conditions in the detector given by attachments 30-33.

17.31 With a multimeter it was found that there is continuity between the snout and the frame.

18.50 We tried to disconnect the cables of the short side of the bPlast detector and all of the grounds. This seemed to show an open line OL on the multimeter and whence connecting the bPlast grounds back excluding the short side ribbon cable grounds, the multimeter still read OL. 
      We also tried to wedge paper between the short side ribbon cable of bPlast and the snout, but this did not work, the detector still reading continuity between the frame, booster board and snout. 
      The results after booting up the detector again are given by the attachments 34-36. 
      The noise condition is appreciably better than before with 12 out of the 16 FEE64 modules with sub 20 kHz rates.
      Disconnect the BB7 preamp. ground from the frame.

18.54 We powered up bPlast. The noise condition three FEEs got worse, aida01, aida11 and 06. The problem may be associated with downstream grounding. The results are given by the attachments 37-43. 10 out of the 16 FEE64 modules showed sub 20 kHz rates.

19.07 We turned off the power supply on the base of the snout support. PN300 Attachment 23. Rates did not change - attachment 44.

19.10 We turned off the mesytec PSU that is located in the bPlast NIM crate. No change was observed in the rates - attachment 45.

19.13 We turned off the R&SRMP4040 PSU that is located above the AIDA crate. And the noise situation did not change attachment 46.

19.22 Leaving the R&SRMP4040 PSU off we turned back on the mesytec PSU and PN300 PSU, attachment 47.

Summary:

It is clear that with no continuity on the short side we are able to reduce the noise back to the scenario where bPlast was not connected. However, powering bPlast introduced substantial noise in both DSSSDs that did not go away when turning all the PSUs off. There might be some hysteresis in the system (this is just speculation).

The situation is still fair considering that 10 out of the 16 FEE64 modules are in the sub 20 kHz rate level (good noise condition).

TO-DO for 11.04.2024

- Try bringing bPlast ground back to the PSU ground.

- Recheck the downstream detector bias and ground scheme.
       
Attachment 1: Screenshot_from_2024-04-10_09-43-13.png
Screenshot_from_2024-04-10_09-43-13.png
Attachment 2: Screenshot_from_2024-04-10_09-46-53.png
Screenshot_from_2024-04-10_09-46-53.png
Attachment 3: Screenshot_from_2024-04-10_09-49-03.png
Screenshot_from_2024-04-10_09-49-03.png
Attachment 4: Screenshot_from_2024-04-10_09-49-35.png
Screenshot_from_2024-04-10_09-49-35.png
Attachment 5: Screenshot_from_2024-04-10_09-50-23.png
Screenshot_from_2024-04-10_09-50-23.png
Attachment 6: Screenshot_from_2024-04-10_09-50-52.png
Screenshot_from_2024-04-10_09-50-52.png
Attachment 7: Screenshot_from_2024-04-10_09-50-42.png
Screenshot_from_2024-04-10_09-50-42.png
Attachment 8: Screenshot_from_2024-04-10_09-53-41.png
Screenshot_from_2024-04-10_09-53-41.png
Attachment 9: Screenshot_from_2024-04-10_09-53-17.png
Screenshot_from_2024-04-10_09-53-17.png
Attachment 10: Screenshot_from_2024-04-10_09-52-49.png
Screenshot_from_2024-04-10_09-52-49.png
Attachment 11: Screenshot_from_2024-04-10_09-52-22.png
Screenshot_from_2024-04-10_09-52-22.png
Attachment 12: Screenshot_from_2024-04-10_09-59-02.png
Screenshot_from_2024-04-10_09-59-02.png
Attachment 13: Screenshot_from_2024-04-10_10-51-20.png
Screenshot_from_2024-04-10_10-51-20.png
Attachment 14: Screenshot_from_2024-04-10_10-50-37.png
Screenshot_from_2024-04-10_10-50-37.png
Attachment 15: Screenshot_from_2024-04-10_10-52-40.png
Screenshot_from_2024-04-10_10-52-40.png
Attachment 16: Screenshot_from_2024-04-10_10-52-16.png
Screenshot_from_2024-04-10_10-52-16.png
Attachment 17: Screenshot_from_2024-04-10_11-41-42.png
Screenshot_from_2024-04-10_11-41-42.png
Attachment 18: IMG_0010.JPG
IMG_0010.JPG
Attachment 19: IMG_0007.JPG
IMG_0007.JPG
Attachment 20: IMG_0012.JPG
IMG_0012.JPG
Attachment 21: IMG_0005.JPG
IMG_0005.JPG
Attachment 22: IMG_0009.JPG
IMG_0009.JPG
Attachment 23: IMG_0015.JPG
IMG_0015.JPG
Attachment 24: IMG_0014.JPG
IMG_0014.JPG
Attachment 25: IMG_0017.JPG
IMG_0017.JPG
Attachment 26: IMG_0006.JPG
IMG_0006.JPG
Attachment 27: IMG_0013.JPG
IMG_0013.JPG
Attachment 28: Screenshot_from_2024-04-10_13-18-09.png
Screenshot_from_2024-04-10_13-18-09.png
Attachment 29: Screenshot_from_2024-04-10_15-55-15.png
Screenshot_from_2024-04-10_15-55-15.png
Attachment 30: Screenshot_from_2024-04-10_17-19-01.png
Screenshot_from_2024-04-10_17-19-01.png
Attachment 31: Screenshot_from_2024-04-10_17-13-25.png
Screenshot_from_2024-04-10_17-13-25.png
Attachment 32: Screenshot_from_2024-04-10_17-12-33.png
Screenshot_from_2024-04-10_17-12-33.png
Attachment 33: Screenshot_from_2024-04-10_17-04-11.png
Screenshot_from_2024-04-10_17-04-11.png
Attachment 34: Screenshot_from_2024-04-10_18-49-54.png
Screenshot_from_2024-04-10_18-49-54.png
Attachment 35: Screenshot_from_2024-04-10_18-49-50.png
Screenshot_from_2024-04-10_18-49-50.png
Attachment 36: Screenshot_from_2024-04-10_18-49-44.png
Screenshot_from_2024-04-10_18-49-44.png
Attachment 37: Screenshot_from_2024-04-10_19-00-27.png
Screenshot_from_2024-04-10_19-00-27.png
Attachment 38: Screenshot_from_2024-04-10_18-59-41.png
Screenshot_from_2024-04-10_18-59-41.png
Attachment 39: Screenshot_from_2024-04-10_18-59-26.png
Screenshot_from_2024-04-10_18-59-26.png
Attachment 40: Screenshot_from_2024-04-10_18-58-43.png
Screenshot_from_2024-04-10_18-58-43.png
Attachment 41: Screenshot_from_2024-04-10_18-58-21.png
Screenshot_from_2024-04-10_18-58-21.png
Attachment 42: Screenshot_from_2024-04-10_18-57-53.png
Screenshot_from_2024-04-10_18-57-53.png
Attachment 43: Screenshot_from_2024-04-10_18-57-46.png
Screenshot_from_2024-04-10_18-57-46.png
Attachment 44: Screenshot_from_2024-04-10_19-23-16.png
Screenshot_from_2024-04-10_19-23-16.png
Attachment 45: Screenshot_from_2024-04-10_19-20-55.png
Screenshot_from_2024-04-10_19-20-55.png
Attachment 46: Screenshot_from_2024-04-10_19-12-22.png
Screenshot_from_2024-04-10_19-12-22.png
Attachment 47: Screenshot_from_2024-04-10_19-09-07.png
Screenshot_from_2024-04-10_19-09-07.png
  569   Tue Apr 9 13:22:41 2024 TD, JB, CCContinued of 9th of April

Disconnected test - pulser from aida08. Noise conditions improved slightly, probably less pulser events. (Attachments 1 & 2).

We recabled the dew point sensor and flow sensor for the water supply interlock to the FEE64s. When we tried to power up the interlock box, the PSU (charging block) LED was blinking indicating an internal power instability. We then plugged each of the sensors out and into the interlock box and found that the flow sensor was the source of the power supply issue.

The HDMI cable of aida04 appears to be broken or wobbly, but the signals still look the same after powering up again. The adaptor board was reseated.

Pulser peak of test - remeasured the peak in aida02 1.8.L and found the same result as before with 118 channels FWHM c. 95-100 keV FWHM energy resolution. (Attachment 3)

bPlast booster boards connected caused dramatic increase in the noise observed in AIDA. Grounding still needs to take place.

TO-DO for 10.04 - 11.04:

- aida04 asic #1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?

- noise optimisation of aida-bplast -- (friday if BB7 also powered).

 

Attachment 1: Screenshot_from_2024-04-09_14-22-20.png
Screenshot_from_2024-04-09_14-22-20.png
Attachment 2: Screenshot_from_2024-04-09_14-22-16.png
Screenshot_from_2024-04-09_14-22-16.png
Attachment 3: Screenshot_from_2024-04-09_17-18-45.png
Screenshot_from_2024-04-09_17-18-45.png
  568   Tue Apr 9 09:04:07 2024 TD[How To] Remote control of BNC PB-5
BNC PB-5 set to remote control

Login to RPi nnrpi2

ssh -X nnrpi2 -l pi

Start PB-5 app - attachment 1

./BNC-PB-5/pb5.py
Attachment 1: Screenshot_from_2024-04-09_10-03-42.png
Screenshot_from_2024-04-09_10-03-42.png
  567   Tue Apr 9 08:37:47 2024 TDTuesday 9 April
09.36 Cooling water temperature and flow OK

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      info code 4, correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 5-10

      per FEE64 Rate spectra - attachments 11

      per 1.8.W spectra - 20us FSR - attachments 12-13

12.25 DSSSD bias & leakage current OK - attachment 16

      FEE64 temps OK - attachment 15
      *except* aida02 ASIC temp which is known to be u/s

      ADC data item stats - attachments 14

      system stable
Attachment 1: Screenshot_from_2024-04-09_09-50-58.png
Screenshot_from_2024-04-09_09-50-58.png
Attachment 2: Screenshot_from_2024-04-09_09-51-07.png
Screenshot_from_2024-04-09_09-51-07.png
Attachment 3: Screenshot_from_2024-04-09_09-51-26.png
Screenshot_from_2024-04-09_09-51-26.png
Attachment 4: Screenshot_from_2024-04-09_09-51-56.png
Screenshot_from_2024-04-09_09-51-56.png
Attachment 5: Screenshot_from_2024-04-09_09-53-31.png
Screenshot_from_2024-04-09_09-53-31.png
Attachment 6: Screenshot_from_2024-04-09_09-53-19.png
Screenshot_from_2024-04-09_09-53-19.png
Attachment 7: Screenshot_from_2024-04-09_09-52-58.png
Screenshot_from_2024-04-09_09-52-58.png
Attachment 8: Screenshot_from_2024-04-09_09-52-48.png
Screenshot_from_2024-04-09_09-52-48.png
Attachment 9: Screenshot_from_2024-04-09_09-52-38.png
Screenshot_from_2024-04-09_09-52-38.png
Attachment 10: Screenshot_from_2024-04-09_09-52-25.png
Screenshot_from_2024-04-09_09-52-25.png
Attachment 11: Screenshot_from_2024-04-09_09-55-57.png
Screenshot_from_2024-04-09_09-55-57.png
Attachment 12: Screenshot_from_2024-04-09_09-54-17.png
Screenshot_from_2024-04-09_09-54-17.png
Attachment 13: Screenshot_from_2024-04-09_09-53-45.png
Screenshot_from_2024-04-09_09-53-45.png
Attachment 14: Screenshot_from_2024-04-09_12-24-55.png
Screenshot_from_2024-04-09_12-24-55.png
Attachment 15: Screenshot_from_2024-04-09_12-24-43.png
Screenshot_from_2024-04-09_12-24-43.png
Attachment 16: Screenshot_from_2024-04-09_12-24-38.png
Screenshot_from_2024-04-09_12-24-38.png
  566   Mon Apr 8 16:39:00 2024 JB, CC, TDMonday 8 April
17.32 Power and detector bias cycle

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2
       *except* aida02 ASIC temp u/s

      ASIC settings 2024Mar27-11.25.32
       slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)

      BNC PB-5 pulser
      amplitude 10.0V
      attenuation x10
      frequency 25Hz
      polarity -
      tau_d 1ms
      tail pulse

      test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

      test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

      All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10

      WR timestamps OK - attachment 11

      ADC data item stats - attachment 3
       12 of 16 < 20k, all < 100k

      per FEE64 Rate spectra - attachment 4

      per n+n FEE64 1.8.L spectra - attachment 5

      1.8.W spectra - 20us FSR - attachments 6-9
       preamplifier output noise generally very good

*Current ground configuration*

CAEN N1419ET LK fitted (non floating outputs)

ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15

ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11

LK3 fitted aida03, aida07

LK1 fitted aida02, aida04, aid06, aida08

test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs

JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )

AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560

Snout configuration:

- all bPlast cables are disconnected and floating. 
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.


To Do:

- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.

- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?




       
Attachment 1: Screenshot_from_2024-04-08_17-32-49.png
Screenshot_from_2024-04-08_17-32-49.png
Attachment 2: Screenshot_from_2024-04-08_17-33-10.png
Screenshot_from_2024-04-08_17-33-10.png
Attachment 3: Screenshot_from_2024-04-08_17-33-14.png
Screenshot_from_2024-04-08_17-33-14.png
Attachment 4: Screenshot_from_2024-04-08_17-33-33.png
Screenshot_from_2024-04-08_17-33-33.png
Attachment 5: Screenshot_from_2024-04-08_17-34-40.png
Screenshot_from_2024-04-08_17-34-40.png
Attachment 6: Screenshot_from_2024-04-08_17-35-00.png
Screenshot_from_2024-04-08_17-35-00.png
Attachment 7: Screenshot_from_2024-04-08_17-35-19.png
Screenshot_from_2024-04-08_17-35-19.png
Attachment 8: Screenshot_from_2024-04-08_17-37-28.png
Screenshot_from_2024-04-08_17-37-28.png
Attachment 9: Screenshot_from_2024-04-08_17-37-56.png
Screenshot_from_2024-04-08_17-37-56.png
Attachment 10: Screenshot_from_2024-04-08_17-48-23.png
Screenshot_from_2024-04-08_17-48-23.png
Attachment 11: Screenshot_from_2024-04-08_17-48-49.png
Screenshot_from_2024-04-08_17-48-49.png
  565   Mon Apr 8 11:18:19 2024 TDS505 offline analysis data file R3_150
DRG quotes ( from Elog ) S505 FEE64 configuration as

"Configuration: Single/Narrow - 8 FEEs, 2 DSSDs

DSSD 1:
 X 0-63    =  3 (HV -ve)
 X 64-127  =  1
 Y 0-63    =  4 (HV 0v) 
 Y 64-127  =  2 
DSSD 2:
 X 0-63    =  7 (HV -ve)
 X 64-127  =  5
 Y 0-63    =  8 (HV 0v)
 Y 64-127  =  6 


Scalers:
1 - Pulser
2 - n/c
3 - Time Machine Original
4 - Time Machine Delayed
5 - n/c
6 - n/c
7 - SC41 L
8 - SC41 R"

LEC (20MeV FSR) spectra

attachment 1 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 2 E_p versus E_n DSSSD#1 & DSSSD#2

HEC (20GeV FSR) spectra

attachment 3 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 4 E_p versus E_n DSSSD#1 & DSSSD#2

attachment variables.dat ( ADC offsets, e_diff windows & FEE64 onfiguration ) NAMELIST I/O format
Attachment 1: Screenshot_from_2024-03-29_20-21-07.png
Screenshot_from_2024-03-29_20-21-07.png
Attachment 2: Screenshot_from_2024-03-29_20-21-45.png
Screenshot_from_2024-03-29_20-21-45.png
Attachment 3: Screenshot_from_2024-03-29_20-22-29.png
Screenshot_from_2024-03-29_20-22-29.png
Attachment 4: Screenshot_from_2024-03-29_20-23-06.png
Screenshot_from_2024-03-29_20-23-06.png
Attachment 5: variables.dat
  $variables
  offset(   0) =    28.32
  offset(   1) =    17.71
  offset(   2) =    -9.75
  offset(   3) =    -5.24
  offset(   4) =   -10.84
  offset(   5) =     0.13
  offset(   6) =    28.59
  offset(   7) =    28.54
  offset(   8) =     0.83
  offset(   9) =   -12.15
  offset(  10) =    29.36
  offset(  11) =   -50.47
  offset(  12) =    10.03
  offset(  13) =   -26.90
  offset(  14) =    12.49
  offset(  15) =    10.57
  offset(  16) =     9.29
  offset(  17) =     1.09
  offset(  18) =    21.86
  offset(  19) =    -0.32
  offset(  20) =    22.59
  offset(  21) = -9999.99
  offset(  22) =   -20.22
  offset(  23) =     6.61
  offset(  24) =    -8.80
  offset(  25) =    -1.15
  offset(  26) =    13.08
  offset(  27) =    -9.03
  offset(  28) =    -2.67
  offset(  29) =    -9.42
  offset(  30) =     8.07
  offset(  31) =    19.01
  offset(  32) =     6.86
  offset(  33) =    15.16
  offset(  34) =    -3.89
  offset(  35) =   -18.80
  offset(  36) =    17.79
  offset(  37) =    16.45
  offset(  38) =    -0.74
  offset(  39) =     0.69
  offset(  40) =   -14.59
  offset(  41) =   -10.02
  offset(  42) =    11.38
  offset(  43) =    -2.57
  offset(  44) =    15.48
  offset(  45) =    14.57
  offset(  46) =     2.95
  offset(  47) =    30.70
  offset(  48) =   -12.55
  offset(  49) =   -15.61
  offset(  50) =    -6.61
  offset(  51) =    -7.10
  offset(  52) =   -33.20
  offset(  53) =    21.42
  offset(  54) =    -4.07
  offset(  55) =   -18.23
  offset(  56) =     8.83
  offset(  57) =   -22.74
  offset(  58) =    11.47
  offset(  59) =    10.67
  offset(  60) =   -28.65
  offset(  61) =    15.29
  offset(  62) =     5.76
  offset(  63) =   -11.57
  offset(  64) =     1.95
  offset(  65) = -9999.99
  offset(  66) =    22.52
  offset(  67) =    42.10
  offset(  68) =    -0.20
  offset(  69) =     9.78
  offset(  70) =   -11.48
  offset(  71) =    -0.03
  offset(  72) =   -12.50
  offset(  73) =    25.08
  offset(  74) =    12.77
  offset(  75) =    -2.76
  offset(  76) =     7.90
  offset(  77) =     9.85
  offset(  78) =     5.20
  offset(  79) =   -13.91
  offset(  80) =    18.82
  offset(  81) =    -8.25
  offset(  82) =    -8.40
  offset(  83) =    13.67
  offset(  84) =   -10.37
  offset(  85) =    28.32
  offset(  86) =    11.42
  offset(  87) =     2.74
  offset(  88) =   -24.88
  offset(  89) =    11.28
  offset(  90) =   -26.69
  offset(  91) =     0.91
  offset(  92) =     8.04
  offset(  93) =     7.64
  offset(  94) =    10.10
  offset(  95) =   -20.71
  offset(  96) =     6.72
  offset(  97) =   -12.63
  offset(  98) =    10.27
  offset(  99) =    16.05
  offset( 100) =     8.96
  offset( 101) =    33.57
  offset( 102) =    28.81
  offset( 103) =    23.49
  offset( 104) =     7.96
  offset( 105) =     6.07
  offset( 106) =    21.57
  offset( 107) =    13.06
  offset( 108) =     0.82
  offset( 109) =    -3.55
  offset( 110) =    -5.21
  offset( 111) =    11.48
  offset( 112) =    16.21
  offset( 113) =    -0.10
  offset( 114) =     3.87
  offset( 115) =    19.76
  offset( 116) =    -9.11
  offset( 117) =    24.64
  offset( 118) =     1.82
  offset( 119) = -9999.99
  offset( 120) =    22.80
  offset( 121) =    -7.37
  offset( 122) =   -18.32
  offset( 123) =   -24.74
  offset( 124) =    -4.71
  offset( 125) =    19.64
  offset( 126) =    53.61
  offset( 127) =     6.57
  offset( 128) =     3.21
  offset( 129) =     7.20
  offset( 130) =     5.55
  offset( 131) =    -3.40
  offset( 132) =   -16.40
  offset( 133) =    -1.71
  offset( 134) =   -14.85
  offset( 135) =   -29.53
  offset( 136) =     0.68
  offset( 137) =    -0.41
  offset( 138) =    17.59
  offset( 139) =   -28.92
  offset( 140) =    -8.79
  offset( 141) =    11.69
  offset( 142) =    -5.80
  offset( 143) =    -8.21
  offset( 144) =    13.90
  offset( 145) =    11.27
  offset( 146) =   -17.02
  offset( 147) =    -8.19
  offset( 148) =     6.14
  offset( 149) =   -13.13
  offset( 150) =    15.41
  offset( 151) =     0.12
  offset( 152) =    -5.02
  offset( 153) =     4.25
  offset( 154) =     3.84
  offset( 155) =    13.34
  offset( 156) =    16.79
  offset( 157) =   -36.21
  offset( 158) =    -2.18
  offset( 159) =    12.94
  offset( 160) =     2.05
  offset( 161) =   -17.31
  offset( 162) =    18.38
  offset( 163) =    -7.17
  offset( 164) =     6.59
  offset( 165) =   -13.36
  offset( 166) =   -12.11
  offset( 167) =    -3.91
  offset( 168) =    10.97
  offset( 169) =   -25.72
  offset( 170) =   -11.53
  offset( 171) =     1.94
  offset( 172) =   -12.76
  offset( 173) =    18.55
  offset( 174) =    13.49
  offset( 175) =    -9.45
  offset( 176) =   -10.19
  offset( 177) =    18.69
  offset( 178) =    24.49
  offset( 179) =    30.44
  offset( 180) =    41.95
  offset( 181) =     1.55
  offset( 182) =    20.88
  offset( 183) =     4.76
  offset( 184) =    15.80
  offset( 185) =    12.37
  offset( 186) =   -13.03
  offset( 187) =    15.64
  offset( 188) =    -8.44
  offset( 189) =    -3.02
  offset( 190) =    10.57
  offset( 191) =     6.88
  offset( 192) =   -17.81
  offset( 193) =    -5.77
  offset( 194) =   -18.29
  offset( 195) =    17.62
  offset( 196) =   -30.66
  offset( 197) =    -7.37
  offset( 198) =    36.32
  offset( 199) =     3.42
  offset( 200) =   -21.92
  offset( 201) =    15.88
  offset( 202) =   -17.33
  offset( 203) =    -2.71
  offset( 204) =   -21.06
  offset( 205) =    -4.53
  offset( 206) =    -1.59
  offset( 207) =    -9.54
  offset( 208) =    -0.04
  offset( 209) =    23.57
  offset( 210) =    13.83
  offset( 211) =    -9.28
  offset( 212) =    12.42
  offset( 213) =    23.85
  offset( 214) =     7.90
  offset( 215) =    -7.29
  offset( 216) =    26.18
  offset( 217) =    33.05
  offset( 218) =    32.10
  offset( 219) =    -8.10
  offset( 220) =    16.16
  offset( 221) =     3.94
  offset( 222) =   -18.22
  offset( 223) =   -16.84
  offset( 224) =     5.79
  offset( 225) =    21.07
  offset( 226) =    45.18
  offset( 227) =   -13.85
  offset( 228) =     6.76
  offset( 229) =    13.59
  offset( 230) =     8.28
  offset( 231) =    -9.32
  offset( 232) =    21.30
  offset( 233) =     7.02
  offset( 234) =    24.11
  offset( 235) =    16.26
  offset( 236) =    11.46
  offset( 237) =     2.16
  offset( 238) =     7.19
  offset( 239) =    18.45
  offset( 240) =     6.65
  offset( 241) =     6.56
  offset( 242) =   -33.59
  offset( 243) =    14.71
  offset( 244) =    21.28
  offset( 245) =    -0.73
  offset( 246) =   -10.29
  offset( 247) =    28.50
  offset( 248) =   -11.37
  offset( 249) =   -18.92
  offset( 250) =    23.12
  offset( 251) =     8.87
  offset( 252) =     9.64
  offset( 253) =    16.43
  offset( 254) =    -2.70
  offset( 255) =    26.14
  offset( 256) =    -4.99
  offset( 257) =   -10.88
  offset( 258) =    27.41
  offset( 259) =    -3.48
  offset( 260) =   -11.18
  offset( 261) =    12.60
  offset( 262) =    22.19
  offset( 263) =   -20.16
  offset( 264) =    20.32
  offset( 265) =     9.66
  offset( 266) =    13.30
  offset( 267) =   -12.64
  offset( 268) =    53.69
  offset( 269) =    23.15
  offset( 270) =    43.60
  offset( 271) =    48.58
  offset( 272) =   -13.12
  offset( 273) =   -12.57
  offset( 274) =    35.19
  offset( 275) =    14.84
  offset( 276) =    33.05
  offset( 277) =    13.73
  offset( 278) =    45.45
  offset( 279) =    55.49
  offset( 280) =    48.13
  offset( 281) =     3.86
  offset( 282) =    28.44
  offset( 283) =    23.73
  offset( 284) =    11.53
  offset( 285) =   -19.46
  offset( 286) =    10.81
  offset( 287) =    17.23
  offset( 288) =   -13.85
  offset( 289) =    30.17
  offset( 290) =    10.70
  offset( 291) =    28.06
  offset( 292) =   -14.35
  offset( 293) =    23.46
  offset( 294) =    15.16
  offset( 295) =   -25.15
  offset( 296) =   -15.03
  offset( 297) =   -24.74
  offset( 298) =    -5.94
... 228 more lines ...
  564   Mon Apr 8 10:23:16 2024 TD, JBAIDA Noise optimisation

In the morning we reinstalled FEE64s aida06, aida13, aida07. The rails of the FEE64 boards were found to be incorrect (that of the rails that were used in CARME).

After replacing the rails on the abovementioned FEE64s we powercycled AIDA and found the following results in the histograms and statistics.

As can be seen in attachement 4, aida16 has a noisy waveform and we are going to investigate this further.

When setting the threshold to 0x64 (1 MeV) on the slow comparator threshold we saw that the low energy branch of aida04 was still noisy. (Attachment 6)

- Setting the slow comparator threshold to 0x32 (0.5 MeV) introduced noise into aida08, aida16 and substantially more noise in aida04. (Attachment 7)

- Setting the slow comparator threshold to 0x14 (0.2 MeV) introduced noise into aida11 and substantially more noise in aida04, aida08, aida16 and aida06. (Attachment 8)

- Setting the slow comparator threshold to 0xf (0.15 MeV) introduced more noise into the same aforementioned channels aida04, aida06, aida08, aida11, aida16 (additionally aida02), barring hot strips. (Attachment 9)

 

We covered the end of the snout with a black cloth and this reduced the noise in some of the channels (Attachment 10 and 11).

- We covered the end of the snout with two layers of aluminium foil - this produced the same result as covering the snout with the black cloth (Attachment 12, 13, 14 & 15).

We resumed after lunch.

- Replaced the aida16 adaptor PCB with a spare, and this produced a much more sensible rate and hitpattern in aida16, which could lead us to believe that the aida adapter card was faulty or misaligned. (Attachment 16, 17, 18 & 19).

- Placed two LK1 back on the n+n side of both DSSSDs on aida06 and aida02. The noise and rate did not change at all. (Attachment 20 & 21)

Pulser tests:

- The test + pulser was connected to the p+n sides (top and bottom) at 25 Hz.

- After checking the ASIC control 2 times over, the pulser seems to be ok for all p+n channels excluding aida01, aida13 and aida14. (Attachment 23)

- The pulser peak width of aida09 1.8.L was found to be c. 55 channels FWHM on uncalibrated scale which corresponds to 35 - 40 keV FWHM in energy resolution. (Attachment 24)

- After reseating the adaptor board PCBs we found the same result as before reseating the adaptor boards (Attachment 25).

- Connecting the test - pulser to the n+n  side we found that most of the ASICs behaved normally which ASIC 1 in aida04 was showing abnormal behaviour. (Attachment 26, 28). The test + pulser loop is still in place.

-  The pulser peak width of aida02 1.8.L was found to be c. 118 channels FWHM on uncalibrated scale which corresponds to 95 - 100 keV FWHM in energy resolution. (Attachment 27)

- We set the thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an addtional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

 

Summary:

- From the pulser test it is clear that aida14 ASICs 1&2 are not in good shape and appear to be faulty. 

- We managed to improve from 1 aida module in the sub 20 kHz rate range to 6 aida modules in the sub 20 kHz rate range by the end of the tests.

- We set the slow comparator thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an additional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

 

Attachment 1: Screenshot_from_2024-04-08_11-28-11.png
Screenshot_from_2024-04-08_11-28-11.png
Attachment 2: Screenshot_from_2024-04-08_11-27-25.png
Screenshot_from_2024-04-08_11-27-25.png
Attachment 3: Screenshot_from_2024-04-08_11-26-38.png
Screenshot_from_2024-04-08_11-26-38.png
Attachment 4: Screenshot_from_2024-04-08_11-26-31.png
Screenshot_from_2024-04-08_11-26-31.png
Attachment 5: Screenshot_from_2024-04-08_11-25-57.png
Screenshot_from_2024-04-08_11-25-57.png
Attachment 6: Screenshot_from_2024-04-08_11-31-34.png
Screenshot_from_2024-04-08_11-31-34.png
Attachment 7: Screenshot_from_2024-04-08_11-34-40.png
Screenshot_from_2024-04-08_11-34-40.png
Attachment 8: Screenshot_from_2024-04-08_11-36-50.png
Screenshot_from_2024-04-08_11-36-50.png
Attachment 9: Screenshot_from_2024-04-08_11-40-33.png
Screenshot_from_2024-04-08_11-40-33.png
Attachment 10: Screenshot_from_2024-04-08_11-51-46.png
Screenshot_from_2024-04-08_11-51-46.png
Attachment 11: Screenshot_from_2024-04-08_11-51-50.png
Screenshot_from_2024-04-08_11-51-50.png
Attachment 12: Screenshot_from_2024-04-08_12-09-33.png
Screenshot_from_2024-04-08_12-09-33.png
Attachment 13: Screenshot_from_2024-04-08_12-09-37.png
Screenshot_from_2024-04-08_12-09-37.png
Attachment 14: Screenshot_from_2024-04-08_12-10-20.png
Screenshot_from_2024-04-08_12-10-20.png
Attachment 15: Screenshot_from_2024-04-08_12-10-57.png
Screenshot_from_2024-04-08_12-10-57.png
Attachment 16: Screenshot_from_2024-04-08_15-09-34.png
Screenshot_from_2024-04-08_15-09-34.png
Attachment 17: Screenshot_from_2024-04-08_15-09-08.png
Screenshot_from_2024-04-08_15-09-08.png
Attachment 18: Screenshot_from_2024-04-08_14-45-49.png
Screenshot_from_2024-04-08_14-45-49.png
Attachment 19: Screenshot_from_2024-04-08_14-45-05.png
Screenshot_from_2024-04-08_14-45-05.png
Attachment 20: Screenshot_from_2024-04-08_14-44-19.png
Screenshot_from_2024-04-08_14-44-19.png
Attachment 21: Screenshot_from_2024-04-08_14-44-14.png
Screenshot_from_2024-04-08_14-44-14.png
Attachment 22: Screenshot_from_2024-04-08_16-03-19.png
Screenshot_from_2024-04-08_16-03-19.png
Attachment 23: Screenshot_from_2024-04-08_16-07-48.png
Screenshot_from_2024-04-08_16-07-48.png
Attachment 24: Screenshot_from_2024-04-08_16-13-53.png
Screenshot_from_2024-04-08_16-13-53.png
Attachment 25: Screenshot_from_2024-04-08_16-38-15.png
Screenshot_from_2024-04-08_16-38-15.png
Attachment 26: Screenshot_from_2024-04-08_16-48-25.png
Screenshot_from_2024-04-08_16-48-25.png
Attachment 27: Screenshot_from_2024-04-08_16-52-17.png
Screenshot_from_2024-04-08_16-52-17.png
Attachment 28: Screenshot_from_2024-04-08_16-50-12.png
Screenshot_from_2024-04-08_16-50-12.png
Attachment 29: Screenshot_from_2024-04-08_17-00-38.png
Screenshot_from_2024-04-08_17-00-38.png
Attachment 30: Screenshot_from_2024-04-08_16-58-25.png
Screenshot_from_2024-04-08_16-58-25.png
  563   Sun Apr 7 18:13:56 2024 TDSunday 7 April contd.
19.10 CAEN N1419ET LK fitted
      power cycle

attachments 1-6
Attachment 1: Screenshot_from_2024-04-07_19-13-19.png
Screenshot_from_2024-04-07_19-13-19.png
Attachment 2: Screenshot_from_2024-04-07_19-12-40.png
Screenshot_from_2024-04-07_19-12-40.png
Attachment 3: Screenshot_from_2024-04-07_19-11-24.png
Screenshot_from_2024-04-07_19-11-24.png
Attachment 4: Screenshot_from_2024-04-07_19-11-09.png
Screenshot_from_2024-04-07_19-11-09.png
Attachment 5: Screenshot_from_2024-04-07_19-10-45.png
Screenshot_from_2024-04-07_19-10-45.png
Attachment 6: Screenshot_from_2024-04-07_19-10-30.png
Screenshot_from_2024-04-07_19-10-30.png
  562   Sun Apr 7 13:08:52 2024 TDFEE64 configuration
FEE64 configuration per https://elog.gsi.de/despec/S100/6

Configuration: Wide - 16 FEEs, 2 DSSD

DSSD 1:
 X 0-63    =  15 
 X 63-127  =  9
 X 128-191 =  3 
 X 192-255 =  1
 X 256-319 =  12 
 X 320-385 =  5
 Y 0-63    =  4 
 Y 63-127  =  2 


DSSD 2:
 X 0-63    =  11 
 X 63-127  =  10
 X 128-191 =  7
 X 192-255 =  14
 X 256-319 =  16 
 X 320-385 =  13
 Y 0-63    =  8
 Y 63-127  =  6 

Scalers:
1 - Pulser
2 - FRS Accepted
3 - Time Machine Undelayed
4 - Time Machine Delayed
5 - SC41 L
6 - SC41 R


Thresholds (Slow / Fast / Implant)
DSSD 1 p+n : 0xA (100 keV) / 0xFF (disabled) / 0x02 (200 MeV)
DSSD 1 n+n : 0xF (150 keV) / 0xFF (disabled) / 0x02 (200 MeV)
Attachment 1: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
  561   Sun Apr 7 12:16:27 2024 TDaida06 system console log
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:07:47|
Finished copying zImage to RAM 
07:04:24/13:07:48|
Found 0 errors checking kernel image
07:04:24/13:07:49|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:07:49|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:07:49|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:07:49|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:07:49|LMK #0 : regInit[0]=0x80000000
07:04:24/13:07:49|LMK #0 : regInit[1]=0x10070600
07:04:24/13:07:49|LMK #0 : regInit[2]=0x60601
07:04:24/13:07:50|LMK #0 : regInit[3]=0x60602
07:04:24/13:07:50|LMK #0 : regInit[4]=0x60603
07:04:24/13:07:50|LMK #0 : regInit[5]=0x70624
07:04:24/13:07:50|LMK #0 : regInit[6]=0x70605
07:04:24/13:07:50|LMK #0 : regInit[7]=0x70606
07:04:24/13:07:50|LMK #0 : regInit[8]=0x70627
07:04:24/13:07:50|LMK #0 : regInit[9]=0x10000908
07:04:24/13:07:50|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:07:50|LMK #0 : regInit[11]=0x82800B
07:04:24/13:07:50|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:07:50|LMK #0 : regInit[13]=0x830020E
07:04:24/13:07:50|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:07:50|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:07:50|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:07:50|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal not completed 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:07:51|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:07:54|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:08:00|Using Xilinx Virtex440 machine description
07:04:24/13:08:01|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:08:01|Zone PFN ranges:
07:04:24/13:08:01|  DMA      0x00000000 -> 0x00007000
07:04:24/13:08:01|  Normal   0x00007000 -> 0x00007000
07:04:24/13:08:01|Movable zone start PFN for each node
07:04:24/13:08:01|early_node_map[1] active PFN ranges
07:04:24/13:08:01|    0: 0x00000000 -> 0x00007000
07:04:24/13:08:01|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:08:01|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:08:02|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:08:02|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:08:02|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:08:02|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:08:02|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:08:02|Kernel virtual memory layout:
07:04:24/13:08:02|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:08:02|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:08:02|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:08:02|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:08:02|NR_IRQS:512
07:04:24/13:08:02|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:08:02|Console: colour dummy device 80x25
07:04:24/13:08:02|Mount-cache hash table entries: 512
07:04:24/13:08:02|NET: Registered protocol family 16
07:04:24/13:08:02|PCI: Probing PCI hardware
07:04:24/13:08:02|bio: create slab <bio-0> at 0
07:04:24/13:08:02|NET: Registered protocol family 2
07:04:24/13:08:02|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:08:02|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:08:03|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:08:03|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:08:03|TCP reno registered
07:04:24/13:08:03|NET: Registered protocol family 1
07:04:24/13:08:03|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:08:03|msgmni has been set to 214
07:04:24/13:08:03|io scheduler noop registered
07:04:24/13:08:03|io scheduler anticipatory registered
07:04:24/13:08:03|io scheduler deadline registered
07:04:24/13:08:03|io scheduler cfq registered (default)
07:04:24/13:08:03|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:08:03|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:08:03|console [ttyS0] enabled
07:04:24/13:08:03|brd: module loaded
07:04:24/13:08:03|loop: module loaded
07:04:24/13:08:03|Device Tree Probing 'ethernet'
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:08:03|XLlTemac: DCR address: 0x80
07:04:24/13:08:03|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:08:03|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:08:03|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:08:04|XTemac: PHY detected at address 7.
07:04:24/13:08:04|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:08:04|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04|Using buffer write method
07:04:24/13:08:04|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:08:04|cmdlinepart partition parsing not available
07:04:24/13:08:04|RedBoot partition parsing not available
07:04:24/13:08:04|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:08:04|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:08:04|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:08:04|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:08:04|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:08:04|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:08:05|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32766
07:04:24/13:08:05|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32765
07:04:24/13:08:05|mice: PS/2 mouse device common for all mice
07:04:24/13:08:05|Device Tree Probing 'i2c'
07:04:24/13:08:05| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:08:05|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:08:05|TCP cubic registered
07:04:24/13:08:05|NET: Registered protocol family 17
07:04:24/13:08:05|RPC: Registered udp transport module.
07:04:24/13:08:05|RPC: Registered tcp transport module.
07:04:24/13:08:05|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:08:06|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:08:08|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:08:08|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:08:08|Sending DHCP requests .
07:04:24/13:08:10|eth0: XLlTemac: PHY Link carrier lost.
07:04:24/13:08:10|..... timed out!
07:04:24/13:09:39|IP-Config: Reopening network devices...
07:04:24/13:09:39|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:09:40|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:09:42|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:09:42|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:09:42|Sending DHCP requests ...... timed out!
07:04:24/13:11:15|IP-Config: Auto-configuration of network failed.
07:04:24/13:11:15|Root-NFS: No NFS server available, giving up.
07:04:24/13:11:15|VFS: Unable to mount root fs via NFS, trying floppy.
07:04:24/13:11:15|VFS: Cannot open root device "nfs" or unknown-block(2,0)
07:04:24/13:11:15|Please append a correct "root=" boot option; here are the available partitions:
07:04:24/13:11:15|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
07:04:24/13:11:15|Call Trace:
07:04:24/13:11:15|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
07:04:24/13:11:15|[c6827f10] [c00345bc] panic+0x94/0x168
07:04:24/13:11:15|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
07:04:24/13:11:15|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
07:04:24/13:11:15|[c6827fd0] [c0341220] kernel_init+0x104/0x130
07:04:24/13:11:15|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
07:04:24/13:11:15|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:14:15|
Finished copying zImage to RAM 
07:04:24/13:14:16|
Found 0 errors checking kernel image
07:04:24/13:14:17|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:14:17|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:14:17|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:14:17|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:14:17|LMK #0 : regInit[0]=0x80000000
07:04:24/13:14:18|LMK #0 : regInit[1]=0x10070600
07:04:24/13:14:18|LMK #0 : regInit[2]=0x60601
07:04:24/13:14:18|LMK #0 : regInit[3]=0x60602
07:04:24/13:14:18|LMK #0 : regInit[4]=0x60603
07:04:24/13:14:18|LMK #0 : regInit[5]=0x70624
07:04:24/13:14:18|LMK #0 : regInit[6]=0x70605
07:04:24/13:14:18|LMK #0 : regInit[7]=0x70606
07:04:24/13:14:18|LMK #0 : regInit[8]=0x70627
07:04:24/13:14:18|LMK #0 : regInit[9]=0x10000908
07:04:24/13:14:18|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:14:18|LMK #0 : regInit[11]=0x82800B
07:04:24/13:14:18|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:14:18|LMK #0 : regInit[13]=0x830020E
07:04:24/13:14:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:14:18|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:14:18|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:14:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:14:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:14:22|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:14:28|Using Xilinx Virtex440 machine description
07:04:24/13:14:29|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:14:29|Zone PFN ranges:
07:04:24/13:14:29|  DMA      0x00000000 -> 0x00007000
07:04:24/13:14:29|  Normal   0x00007000 -> 0x00007000
07:04:24/13:14:29|Movable zone start PFN for each node
07:04:24/13:14:29|early_node_map[1] active PFN ranges
07:04:24/13:14:29|    0: 0x00000000 -> 0x00007000
07:04:24/13:14:29|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:14:29|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:14:29|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:14:30|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:14:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:14:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:14:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:14:30|Kernel virtual memory layout:
07:04:24/13:14:30|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:14:30|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:14:30|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:14:30|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:14:30|NR_IRQS:512
07:04:24/13:14:30|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:14:30|Console: colour dummy device 80x25
07:04:24/13:14:30|Mount-cache hash table entries: 512
07:04:24/13:14:30|NET: Registered protocol family 16
07:04:24/13:14:30|PCI: Probing PCI hardware
07:04:24/13:14:30|bio: create slab <bio-0> at 0
07:04:24/13:14:30|NET: Registered protocol family 2
07:04:24/13:14:30|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:14:30|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:14:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:14:31|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:14:31|TCP reno registered
07:04:24/13:14:31|NET: Registered protocol family 1
07:04:24/13:14:31|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:14:31|msgmni has been set to 214
07:04:24/13:14:31|io scheduler noop registered
07:04:24/13:14:31|io scheduler anticipatory registered
07:04:24/13:14:31|io scheduler deadline registered
07:04:24/13:14:31|io scheduler cfq registered (default)
07:04:24/13:14:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:14:31|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:14:31|console [ttyS0] enabled
07:04:24/13:14:31|brd: module loaded
07:04:24/13:14:31|loop: module loaded
07:04:24/13:14:31|Device Tree Probing 'ethernet'
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:14:31|XLlTemac: DCR address: 0x80
07:04:24/13:14:31|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:14:31|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:14:31|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:14:32|XTemac: PHY detected at address 7.
07:04:24/13:14:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:14:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32|Using buffer write method
07:04:24/13:14:32|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:14:32|cmdlinepart partition parsing not available
07:04:24/13:14:32|RedBoot partition parsing not available
07:04:24/13:14:32|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:14:32|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:14:32|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:14:32|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:14:32|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:14:32|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:14:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32766
07:04:24/13:14:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32765
07:04:24/13:14:33|mice: PS/2 mouse device common for all mice
07:04:24/13:14:33|Device Tree Probing 'i2c'
07:04:24/13:14:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:14:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:14:33|TCP cubic registered
07:04:24/13:14:33|NET: Registered protocol family 17
07:04:24/13:14:33|RPC: Registered udp transport module.
07:04:24/13:14:33|RPC: Registered tcp transport module.
07:04:24/13:14:33|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:14:34|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:14:36|Sending DHCP requests ., OK
07:04:24/13:14:37|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
07:04:24/13:14:37|IP-Config: Complete:
07:04:24/13:14:37|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
07:04:24/13:14:37|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
07:04:24/13:14:37|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
07:04:24/13:14:37|Looking up port of RPC 100003/2 on 192.168.11.99
07:04:24/13:14:37|Looking up port of RPC 100005/1 on 192.168.11.99
07:04:24/13:14:37|VFS: Mounted root (nfs filesystem) on device 0:12.
07:04:24/13:14:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
07:04:24/13:14:38|Starting sysinit...
07:04:24/13:14:38|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
07:04:24/13:14:39|		Press 'I' to enter interactive startup.
07:04:24/13:14:39|Setting clock  (utc): Sun Apr  7 12:14:39 BST 2024 [  OK  ]
07:04:24/13:14:39|Building the cache [  OK  ]
07:04:24/13:14:39|Setting hostname aida06:  [  OK  ]
07:04:24/13:14:41|Mounting local filesystems:  [  OK  ]
07:04:24/13:14:42|Enabling /etc/fstab swaps:  [  OK  ]
07:04:24/13:14:45|Finishing sysinit...
INIT: Entering runlevel: 3
07:04:24/13:14:48|Entering non-interactive startup
07:04:24/13:14:48|FATAL: Module ipv6 not found.
07:04:24/13:14:49|Bringing up loopback interface:  [  OK  ]
07:04:24/13:14:52|FATAL: Module ipv6 not found.
07:04:24/13:14:53|Starting system logger: [  OK  ]
07:04:24/13:14:54|Starting kernel logger: [  OK  ]
07:04:24/13:14:54|Starting rpcbind: [  OK  ]
07:04:24/13:14:56|Mounting NFS filesystems:  [  OK  ]
07:04:24/13:14:56|Mounting other filesystems:  [  OK  ]
07:04:24/13:14:56|Starting xinetd: [  OK  ]
07:04:24/13:14:57|Starting midas:  Starting MIDAS Data Acquisition for aida06
07:04:24/13:14:58|xaida: device parameters: base=0x81000000 size=0x200000
07:04:24/13:15:03|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
07:04:24/13:15:03|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
07:04:24/13:15:03|xaida: driver assigned major number 254
07:04:24/13:15:04|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
07:04:24/13:15:09|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
07:04:24/13:15:09|AIDAMEM: aidamem: driver assigned major number 253
07:04:24/13:15:09|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
07:04:24/13:15:16|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
07:04:24/13:15:16|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
07:04:24/13:15:16|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
07:04:24/13:15:16|Computer Name = aida06; Temp Directory = /tmp/tcl361
07:04:24/13:15:21|
07:04:24/13:15:25|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
07:04:24/13:15:25|
07:04:24/13:15:25|Built without pthreads
07:04:24/13:15:25|
07:04:24/13:15:25|Creating NetVars
07:04:24/13:15:25|Output buffer length = 65504; format option = 4; transfer option = 3
07:04:24/13:15:25|EB transfer option = 3
07:04:24/13:15:26|NetVars creaxaida: open:
07:04:24/13:15:26|ted and initialised
07:04:24/13:15:26|StatisticsAIDAMEM: aidamem_open:
07:04:24/13:15:26| thread starting
07:04:24/13:15:26|Data Acquisition task has PID 375
07:04:24/13:15:26|Statistics thread created
07:04:24/13:15:26|Stat/Rate creation thread starting
07:04:24/13:15:26|Stat/Rate creation thread created
07:04:24/13:15:26|Hit/Rate creation thread starting
07:04:24/13:15:26|Hit/Rate creation thread created
07:04:24/13:15:26|AIDA Heartbeat thread starting
07:04:24/13:15:26|Heartbeat thread created
07:04:24/13:15:26|Installing signal handlers
07:04:24/13:15:26|Done
07:04:24/13:15:26|ModuleNum = 0
07:04:24/13:15:26|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
07:04:24/13:15:26|package limit is not available: can't find package limit
07:04:24/13:15:26|Running with default file descriptor limit
07:04:24/13:15:26|Spectra table initialised
07:04:24/13:15:26|AIDA Data Acquisition now all ready to start
07:04:24/13:15:26|SIGBUS, SIGSEGV and SIGPIPE traps setup
07:04:24/13:15:26|package setuid is not available: can't find package setuid
07:04:24/13:15:27|Running as user 0 group 0
07:04:24/13:15:27|[  OK  ]
07:04:24/13:15:36|
07:04:24/13:15:36|DENX ELDK version 4.2 build 2008-04-01
07:04:24/13:15:36|Linux 2.6.31 on a ppc
07:04:24/13:15:36|
07:04:24/13:15:36|aida06 login: /debug user
  560   Sun Apr 7 12:02:23 2024 TDSunday 7 April
13.00 Cooling water temperature and flow OK - attachment 1

      Test of AIDA 2x MSL type BB18(DS)-1000 24cm x 8cm DSSSDs 'as is' ( to be defined later in this Elog )

      DSSSD bias & leakage current OK - attachment 2
      Ambient temp 24.7 deg C, d.p. 7.1 deg C, RH 32.4%
      Leakage current c. 6.6uA => 4nA/cm2/100um ( very good )

      FEE64 temps OK - attachment 3
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder status - attachment 4

      WR timestamps OK - attachment 5

      WR (info code 4 & 5), correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 6-12

      per FEE64 Rate spectra - attachments 13-15

      per 1.8.W spectra - 20us FSR - attachments 16-1710129


14.45 FEE64 config check - per https://elog.ph.ed.ac.uk/DESPEC/562


FEE64 # PSU cable #           MAC
1       1                     ?
2       2                     ?
3       3                     ?
4       4                     41 ee 71
5       5                     ?
6       6                     ?
7       7                     f6 5a
8       8                     41 d7 cd
9       ?                     ?
10      10                    41 d0 0e
11      11                    41 ee 0f
12      ?                     ?
13      13                    ?
14      ?                     0d 15
15      15                    ?
16      16                    f6 ed

? = no line of sight

- AIDA FEE64 PSU cabling - see attachment 18

Currently

PSU #1 1-3, 2-4, 9-5, 15-12
PSU #2 14-7, 6-8, 10-13, 11-16

Should be changed to

PSU #1 1-3, 2-4, 9-15, 5-12
PSU #2 14-7, 6-8, 10-11, 13-16


- AIDA snout mount to support frame - attachments 19 & 21
LHS ( looking downstream ) incorrect, RHS OK

- LKs
LK1 fitted aida02, aida04, aida06, aida08
LK3 fitted aida03, aida07, aida01, aida14


Significant gaps in snout ( i.e. possible light leaks ) where bPlas cabling exits snout - attachments 20 & 22

bPlas driver PCBs removed 
bPlas ribbon cables, drain wires and Lemo 00.250 cabling disconnected - some cabling/drain wires touching snout etc


15.15 Adjust bPlas ribbon cables/drain wires/Lemo cables to ensure that none are touching snout or AIDA support frame - significant improvement

      DSSSD bias & leakage current OK - attachment 23

      FEE64 temps OK - attachment 24

      ADC data item stats - attachments 25

      per FEE64 Rate spectra - attachments 26
       6x FEE64s < 20k. all FEE64s < 120k ( cf. 4x FEE64s < 20k, 5x FEE64s > 100k, 1x FEE64 320k earlier - attachment 12 )

      per 1.8.W spectra - 20us FSR - attachments 27-28

17.15

LHS top hat washers
LK1, LK3
PSU power cables
tighten gnd screws
attachments 29-34

Note that some of the newly installed FEE64s have the incorrect rails fitted - FEE64s may ground to AIDA support stand

17.55 CAEN N1419ET LK removed -> floating outputs

18.15 +LK1 restored
attachments 35-40

18.40 +pwwer cycle
attachments 41-46
Attachment 1: 20240407_125941.jpg
20240407_125941.jpg
Attachment 2: Screenshot_from_2024-04-07_13-20-57.png
Screenshot_from_2024-04-07_13-20-57.png
Attachment 3: Screenshot_from_2024-04-07_13-29-54.png
Screenshot_from_2024-04-07_13-29-54.png
Attachment 4: Screenshot_from_2024-04-07_13-32-14.png
Screenshot_from_2024-04-07_13-32-14.png
Attachment 5: Screenshot_from_2024-04-07_13-36-47.png
Screenshot_from_2024-04-07_13-36-47.png
Attachment 6: Screenshot_from_2024-04-07_13-42-52.png
Screenshot_from_2024-04-07_13-42-52.png
Attachment 7: Screenshot_from_2024-04-07_13-42-26.png
Screenshot_from_2024-04-07_13-42-26.png
Attachment 8: Screenshot_from_2024-04-07_13-42-04.png
Screenshot_from_2024-04-07_13-42-04.png
Attachment 9: Screenshot_from_2024-04-07_13-41-37.png
Screenshot_from_2024-04-07_13-41-37.png
Attachment 10: Screenshot_from_2024-04-07_13-41-25.png
Screenshot_from_2024-04-07_13-41-25.png
Attachment 11: Screenshot_from_2024-04-07_13-41-03.png
Screenshot_from_2024-04-07_13-41-03.png
Attachment 12: Screenshot_from_2024-04-07_13-40-35.png
Screenshot_from_2024-04-07_13-40-35.png
Attachment 13: Screenshot_from_2024-04-07_13-47-03.png
Screenshot_from_2024-04-07_13-47-03.png
Attachment 14: Screenshot_from_2024-04-07_13-46-47.png
Screenshot_from_2024-04-07_13-46-47.png
Attachment 15: Screenshot_from_2024-04-07_13-46-39.png
Screenshot_from_2024-04-07_13-46-39.png
Attachment 16: Screenshot_from_2024-04-07_13-53-21.png
Screenshot_from_2024-04-07_13-53-21.png
Attachment 17: Screenshot_from_2024-04-07_13-52-25.png
Screenshot_from_2024-04-07_13-52-25.png
Attachment 18: 20240407_143456.jpg
20240407_143456.jpg
Attachment 19: 20240407_143510.jpg
20240407_143510.jpg
Attachment 20: 20240407_143553.jpg
20240407_143553.jpg
Attachment 21: 20240407_143532.jpg
20240407_143532.jpg
Attachment 22: 20240407_143539.jpg
20240407_143539.jpg
Attachment 23: Screenshot_from_2024-04-07_15-10-39.png
Screenshot_from_2024-04-07_15-10-39.png
Attachment 24: Screenshot_from_2024-04-07_15-10-53.png
Screenshot_from_2024-04-07_15-10-53.png
Attachment 25: Screenshot_from_2024-04-07_15-13-00.png
Screenshot_from_2024-04-07_15-13-00.png
Attachment 26: Screenshot_from_2024-04-07_15-12-49.png
Screenshot_from_2024-04-07_15-12-49.png
Attachment 27: Screenshot_from_2024-04-07_15-11-54.png
Screenshot_from_2024-04-07_15-11-54.png
Attachment 28: Screenshot_from_2024-04-07_15-11-06.png
Screenshot_from_2024-04-07_15-11-06.png
Attachment 29: Screenshot_from_2024-04-07_17-14-23.png
Screenshot_from_2024-04-07_17-14-23.png
Attachment 30: Screenshot_from_2024-04-07_17-13-49.png
Screenshot_from_2024-04-07_17-13-49.png
Attachment 31: Screenshot_from_2024-04-07_17-11-59.png
Screenshot_from_2024-04-07_17-11-59.png
Attachment 32: Screenshot_from_2024-04-07_17-11-47.png
Screenshot_from_2024-04-07_17-11-47.png
Attachment 33: Screenshot_from_2024-04-07_17-11-38.png
Screenshot_from_2024-04-07_17-11-38.png
Attachment 34: Screenshot_from_2024-04-07_17-11-19.png
Screenshot_from_2024-04-07_17-11-19.png
Attachment 35: Screenshot_from_2024-04-07_18-12-38.png
Screenshot_from_2024-04-07_18-12-38.png
Attachment 36: Screenshot_from_2024-04-07_18-12-53.png
Screenshot_from_2024-04-07_18-12-53.png
Attachment 37: Screenshot_from_2024-04-07_18-13-03.png
Screenshot_from_2024-04-07_18-13-03.png
Attachment 38: Screenshot_from_2024-04-07_18-14-51.png
Screenshot_from_2024-04-07_18-14-51.png
Attachment 39: Screenshot_from_2024-04-07_18-14-12.png
Screenshot_from_2024-04-07_18-14-12.png
Attachment 40: Screenshot_from_2024-04-07_18-13-10.png
Screenshot_from_2024-04-07_18-13-10.png
Attachment 41: Screenshot_from_2024-04-07_18-44-32.png
Screenshot_from_2024-04-07_18-44-32.png
Attachment 42: Screenshot_from_2024-04-07_18-44-08.png
Screenshot_from_2024-04-07_18-44-08.png
Attachment 43: Screenshot_from_2024-04-07_18-43-30.png
Screenshot_from_2024-04-07_18-43-30.png
Attachment 44: Screenshot_from_2024-04-07_18-43-21.png
Screenshot_from_2024-04-07_18-43-21.png
Attachment 45: Screenshot_from_2024-04-07_18-43-12.png
Screenshot_from_2024-04-07_18-43-12.png
Attachment 46: Screenshot_from_2024-04-07_18-42-56.png
Screenshot_from_2024-04-07_18-42-56.png
  559   Wed Apr 3 13:02:19 2024 NHMerger for 16 FEEs
Changed /MIDAS/Linux/startup/NewMerger

Change parameters -i and -l in master64 to 16 for 16 FEEs
Update NewMerger Options LinksAvailable to 16, LinksInUse to 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Fix NetVar RunOptions 1 (was 0)

Restart Merger HTTPd, Tape, Merger, MBS Spy

Reset/Setup/Go 

16 Links green and status going, all good?

Bias DSSSDs and turn data transfer on

Merger connected, shows rate and updates... no rate in Tape Server?

.. Oops forget to turn on Output to data storage in merger!

Rate in tape server and to MBS: 7 MB/s

Merger, Tape and MBS working with 16 FEEs
Attachment 1: Screenshot_from_2024-04-03_14-15-02.png
Screenshot_from_2024-04-03_14-15-02.png
Attachment 2: Screenshot_from_2024-04-03_14-15-25.png
Screenshot_from_2024-04-03_14-15-25.png
Attachment 3: Screenshot_from_2024-04-03_14-15-36.png
Screenshot_from_2024-04-03_14-15-36.png
Attachment 4: Screenshot_from_2024-04-03_14-15-45.png
Screenshot_from_2024-04-03_14-15-45.png
Attachment 5: Screenshot_from_2024-04-03_14-15-52.png
Screenshot_from_2024-04-03_14-15-52.png
  558   Wed Apr 3 12:42:57 2024 NHReport aida02 WR errors

The WR error counter for aida02 seems to consantly rise

Tried reseating cable on both ends, no change

However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen

Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence?

Attachment 1: Screenshot_from_2024-04-03_13-43-48.png
Screenshot_from_2024-04-03_13-43-48.png
Attachment 2: Screenshot_from_2024-04-03_13-43-38.png
Screenshot_from_2024-04-03_13-43-38.png
Attachment 3: Screenshot_from_2024-04-03_13-43-58.png
Screenshot_from_2024-04-03_13-43-58.png
Attachment 4: Screenshot_from_2024-04-03_13-44-14.png
Screenshot_from_2024-04-03_13-44-14.png
Attachment 5: Screenshot_from_2024-04-03_13-44-36.png
Screenshot_from_2024-04-03_13-44-36.png
  557   Wed Apr 3 12:09:47 2024 NHReport - aida06 frequently fails to boot first time (PHY error)

When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine

Log file attached, key part (to me) is this:

27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|
Attachment 1: ttyUSB15
27:03:24/14:12:59|
ISOL Version 1.00 Date 9th January 2017
27:03:24/14:12:59|
Flash base address=FC000000
27:03:24/14:12:59|
Set Flash to ASync Mode
27:03:24/14:12:59|
XST_SUCCESS
27:03:24/14:12:59|
Finished copying zImage to RAM 
27:03:24/14:12:59|
27:03:24/14:12:59|
Found 0 errors checking kernel image
27:03:24/14:13:00|VHDL version number 0X03350706 
27:03:24/14:13:00|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
27:03:24/14:13:00|
Starting LMK 3200 setup 
27:03:24/14:13:00|
27:03:24/14:13:00|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
27:03:24/14:13:00|
.... SPI Base Address=0x81400000
27:03:24/14:13:00|
clk_control_reg=0x4
27:03:24/14:13:01|Next step is SPIconfig
27:03:24/14:13:01|
Control 32(0x81400000)=0x180
27:03:24/14:13:01|
SlaveSel(0x81400000)=0x3
27:03:24/14:13:01|
Ctrl(0x81400000)=0xE6
27:03:24/14:13:01|
Ctrl(0x81400000)=0x86
27:03:24/14:13:01|SPIconfig done now to set up the LMK3200 registers
27:03:24/14:13:01|LMK #0 : regInit[0]=0x80000000
27:03:24/14:13:01|LMK #0 : regInit[1]=0x10070600
27:03:24/14:13:01|LMK #0 : regInit[2]=0x60601
27:03:24/14:13:01|LMK #0 : regInit[3]=0x60602
27:03:24/14:13:01|LMK #0 : regInit[4]=0x60603
27:03:24/14:13:01|LMK #0 : regInit[5]=0x70624
27:03:24/14:13:01|LMK #0 : regInit[6]=0x70605
27:03:24/14:13:01|LMK #0 : regInit[7]=0x70606
27:03:24/14:13:01|LMK #0 : regInit[8]=0x70627
27:03:24/14:13:01|LMK #0 : regInit[9]=0x10000908
27:03:24/14:13:01|LMK #0 : regInit[10]=0xA0022A09
27:03:24/14:13:01|LMK #0 : regInit[11]=0x82800B
27:03:24/14:13:01|LMK #0 : regInit[12]=0x28C800D
27:03:24/14:13:01|LMK #0 : regInit[13]=0x830020E
27:03:24/14:13:01|LMK #0 : regInit[14]=0xC800180F
27:03:24/14:13:01|
Calibrate completed at 943 counts
27:03:24/14:13:01|
Setting Clock Control =0x0000000B, to set GOE and sync bit
27:03:24/14:13:01|
Ctrl @ SPIstop (0x81400000)=0x186
27:03:24/14:13:01|
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
27:03:24/14:13:01|
27:03:24/14:13:01|
Finished Clock setup LMK03200
27:03:24/14:13:01|
completed LMK 3200 setup 
27:03:24/14:13:02|
Loaded all four ASICs with default settings 
27:03:24/14:13:02|
Setting the ADCs into calibration mode 
27:03:24/14:13:02|
27:03:24/14:13:02|
Control 32(0x81400400)=0x180
27:03:24/14:13:02|
SlaveSel(0x81400400)=0xFF
27:03:24/14:13:02|
Ctrl(0x81400400)=0xE6
27:03:24/14:13:02|
Ctrl(0x81400400)=0x86
27:03:24/14:13:02|
Init : Config of AD9252 SPI ok
27:03:24/14:13:02|
27:03:24/14:13:02|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
27:03:24/14:13:02|
Cal not completed 
27:03:24/14:13:02|
ADC calibrate failed 
27:03:24/14:13:02|
Jumping to kernel simpleboot...
27:03:24/14:13:02|
27:03:24/14:13:02|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
27:03:24/14:13:02|
Allocating 0x3b78cc bytes for kernel ...
27:03:24/14:13:02|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
27:03:24/14:13:05|
27:03:24/14:13:05|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:13:12|
Finalizing device tree... flat tree at 0xbd1300
27:03:24/14:13:12|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
27:03:24/14:13:12|Using Xilinx Virtex440 machine description
27:03:24/14:13:12|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
27:03:24/14:13:12|Zone PFN ranges:
27:03:24/14:13:12|  DMA      0x00000000 -> 0x00007000
27:03:24/14:13:12|  Normal   0x00007000 -> 0x00007000
27:03:24/14:13:12|Movable zone start PFN for each node
27:03:24/14:13:13|early_node_map[1] active PFN ranges
27:03:24/14:13:13|    0: 0x00000000 -> 0x00007000
27:03:24/14:13:13|MMU: Allocated 1088 bytes of context maps for 255 contexts
27:03:24/14:13:13|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
27:03:24/14:13:13|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:13:13|PID hash table entries: 512 (order: 9, 2048 bytes)
27:03:24/14:13:13|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
27:03:24/14:13:13|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
27:03:24/14:13:13|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
27:03:24/14:13:13|Kernel virtual memory layout:
27:03:24/14:13:13|  * 0xffffe000..0xfffff000  : fixmap
27:03:24/14:13:13|  * 0xfde00000..0xfe000000  : consistent mem
27:03:24/14:13:13|  * 0xfde00000..0xfde00000  : early ioremap
27:03:24/14:13:13|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
27:03:24/14:13:13|NR_IRQS:512
27:03:24/14:13:13|clocksource: timebase mult[a00000] shift[22] registered
27:03:24/14:13:13|Console: colour dummy device 80x25
27:03:24/14:13:13|Mount-cache hash table entries: 512
27:03:24/14:13:13|NET: Registered protocol family 16
27:03:24/14:13:14|PCI: Probing PCI hardware
27:03:24/14:13:14|bio: create slab <bio-0> at 0
27:03:24/14:13:14|NET: Registered protocol family 2
27:03:24/14:13:14|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
27:03:24/14:13:14|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
27:03:24/14:13:14|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
27:03:24/14:13:14|TCP: Hash tables configured (established 4096 bind 4096)
27:03:24/14:13:14|TCP reno registered
27:03:24/14:13:14|NET: Registered protocol family 1
27:03:24/14:13:14|ROMFS MTD (C) 2007 Red Hat, Inc.
27:03:24/14:13:14|msgmni has been set to 214
27:03:24/14:13:14|io scheduler noop registered
27:03:24/14:13:14|io scheduler anticipatory registered
27:03:24/14:13:14|io scheduler deadline registered
27:03:24/14:13:14|io scheduler cfq registered (default)
27:03:24/14:13:14|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
27:03:24/14:13:14|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
27:03:24/14:13:14|console [ttyS0] enabled
27:03:24/14:13:14|brd: module loaded
27:03:24/14:13:14|loop: module loaded
27:03:24/14:13:14|Device Tree Probing 'ethernet'
27:03:24/14:13:14|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
27:03:24/14:13:14|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
27:03:24/14:13:15|XLlTemac: DCR address: 0x80
27:03:24/14:13:15|XLlTemac: buffer descriptor size: 32768 (0x8000)
27:03:24/14:13:15|XLlTemac: Allocating DMA descriptors with kmalloc
27:03:24/14:13:15|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
27:03:24/14:13:15|XTemac: PHY detected at address 7.
27:03:24/14:13:15|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
27:03:24/14:13:15|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15|Using buffer write method
27:03:24/14:13:15|cfi_cmdset_0001: Erase suspend on write enabled
27:03:24/14:13:15|cmdlinepart partition parsing not available
27:03:24/14:13:15|RedBoot partition parsing not available
27:03:24/14:13:15|Creating 5 MTD partitions on "fc000000.flash":
27:03:24/14:13:15|0x000000000000-0x000000500000 : "golden_firmware"
27:03:24/14:13:16|0x000000500000-0x000000800000 : "golden_kernel"
27:03:24/14:13:16|0x000000800000-0x000000d00000 : "user_firmware"
27:03:24/14:13:16|0x000000d00000-0x000000fe0000 : "user_kernel"
27:03:24/14:13:16|0x000000fe0000-0x000001000000 : "env_variables"
27:03:24/14:13:16|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
27:03:24/14:13:16|SPI: XIlinx spi: bus number now 32766
27:03:24/14:13:16|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
27:03:24/14:13:16|SPI: XIlinx spi: bus number now 32765
27:03:24/14:13:16|mice: PS/2 mouse device common for all mice
27:03:24/14:13:16|Device Tree Probing 'i2c'
27:03:24/14:13:16| #0 at 0x81600000 mapped to 0xD1030000, irq=22
27:03:24/14:13:16|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
27:03:24/14:13:16|TCP cubic registered
27:03:24/14:13:16|NET: Registered protocol family 17
27:03:24/14:13:16|RPC: Registered udp transport module.
27:03:24/14:13:16|RPC: Registered tcp transport module.
27:03:24/14:13:16|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:13:17|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:13:17|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:13:17|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:13:19|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:13:19|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|
ISOL Version 1.00 Date 9th January 2017
27:03:24/14:26:16|
Flash base address=FC000000
27:03:24/14:26:16|
Set Flash to ASync Mode
27:03:24/14:26:16|
XST_SUCCESS
27:03:24/14:26:16|
Finished copying zImage to RAM 
27:03:24/14:26:17|
27:03:24/14:26:17|
Found 0 errors checking kernel image
27:03:24/14:26:18|VHDL version number 0X03350706 
27:03:24/14:26:18|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
27:03:24/14:26:18|
Starting LMK 3200 setup 
27:03:24/14:26:18|
27:03:24/14:26:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
27:03:24/14:26:18|
.... SPI Base Address=0x81400000
27:03:24/14:26:18|
clk_control_reg=0x4
27:03:24/14:26:18|Next step is SPIconfig
27:03:24/14:26:18|
Control 32(0x81400000)=0x180
27:03:24/14:26:18|
SlaveSel(0x81400000)=0x3
27:03:24/14:26:18|
Ctrl(0x81400000)=0xE6
27:03:24/14:26:18|
Ctrl(0x81400000)=0x86
27:03:24/14:26:18|SPIconfig done now to set up the LMK3200 registers
27:03:24/14:26:18|LMK #0 : regInit[0]=0x80000000
27:03:24/14:26:18|LMK #0 : regInit[1]=0x10070600
27:03:24/14:26:18|LMK #0 : regInit[2]=0x60601
27:03:24/14:26:19|LMK #0 : regInit[3]=0x60602
27:03:24/14:26:19|LMK #0 : regInit[4]=0x60603
27:03:24/14:26:19|LMK #0 : regInit[5]=0x70624
27:03:24/14:26:19|LMK #0 : regInit[6]=0x70605
27:03:24/14:26:19|LMK #0 : regInit[7]=0x70606
27:03:24/14:26:19|LMK #0 : regInit[8]=0x70627
27:03:24/14:26:19|LMK #0 : regInit[9]=0x10000908
27:03:24/14:26:19|LMK #0 : regInit[10]=0xA0022A09
27:03:24/14:26:19|LMK #0 : regInit[11]=0x82800B
27:03:24/14:26:19|LMK #0 : regInit[12]=0x28C800D
27:03:24/14:26:19|LMK #0 : regInit[13]=0x830020E
27:03:24/14:26:19|LMK #0 : regInit[14]=0xC800180F
27:03:24/14:26:19|
Calibrate completed at 943 counts
27:03:24/14:26:19|
Setting Clock Control =0x0000000B, to set GOE and sync bit
27:03:24/14:26:19|
Ctrl @ SPIstop (0x81400000)=0x186
27:03:24/14:26:19|
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
27:03:24/14:26:19|
27:03:24/14:26:19|
Finished Clock setup LMK03200
27:03:24/14:26:19|
completed LMK 3200 setup 
27:03:24/14:26:19|
Loaded all four ASICs with default settings 
27:03:24/14:26:19|
Setting the ADCs into calibration mode 
27:03:24/14:26:19|
27:03:24/14:26:19|
Control 32(0x81400400)=0x180
27:03:24/14:26:19|
SlaveSel(0x81400400)=0xFF
27:03:24/14:26:19|
Ctrl(0x81400400)=0xE6
27:03:24/14:26:19|
Ctrl(0x81400400)=0x86
27:03:24/14:26:19|
Init : Config of AD9252 SPI ok
27:03:24/14:26:19|
27:03:24/14:26:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
27:03:24/14:26:20|
ADCs calibrated
27:03:24/14:26:20|
27:03:24/14:26:20|
Control 32(0x81400400)=0x186
27:03:24/14:26:20|
SlaveSel(0x81400400)=0xFF
27:03:24/14:26:20|
Ctrl(0x81400400)=0xE6
27:03:24/14:26:20|
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
27:03:24/14:26:20|
27:03:24/14:26:20|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
27:03:24/14:26:20|
27:03:24/14:26:20|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
27:03:24/14:26:20|
Allocating 0x3b78cc bytes for kernel ...
27:03:24/14:26:20|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
27:03:24/14:26:23|
27:03:24/14:26:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:26:29|
Finalizing device tree... flat tree at 0xbd1300
27:03:24/14:26:29|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
27:03:24/14:26:30|Using Xilinx Virtex440 machine description
27:03:24/14:26:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
27:03:24/14:26:30|Zone PFN ranges:
27:03:24/14:26:30|  DMA      0x00000000 -> 0x00007000
27:03:24/14:26:30|  Normal   0x00007000 -> 0x00007000
27:03:24/14:26:30|Movable zone start PFN for each node
27:03:24/14:26:30|early_node_map[1] active PFN ranges
27:03:24/14:26:30|    0: 0x00000000 -> 0x00007000
27:03:24/14:26:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
27:03:24/14:26:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
27:03:24/14:26:31|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:26:31|PID hash table entries: 512 (order: 9, 2048 bytes)
27:03:24/14:26:31|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
27:03:24/14:26:31|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
27:03:24/14:26:31|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
27:03:24/14:26:31|Kernel virtual memory layout:
27:03:24/14:26:31|  * 0xffffe000..0xfffff000  : fixmap
27:03:24/14:26:31|  * 0xfde00000..0xfe000000  : consistent mem
27:03:24/14:26:31|  * 0xfde00000..0xfde00000  : early ioremap
27:03:24/14:26:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
27:03:24/14:26:31|NR_IRQS:512
27:03:24/14:26:31|clocksource: timebase mult[a00000] shift[22] registered
27:03:24/14:26:31|Console: colour dummy device 80x25
27:03:24/14:26:31|Mount-cache hash table entries: 512
27:03:24/14:26:31|NET: Registered protocol family 16
27:03:24/14:26:31|PCI: Probing PCI hardware
27:03:24/14:26:31|bio: create slab <bio-0> at 0
27:03:24/14:26:31|NET: Registered protocol family 2
27:03:24/14:26:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
27:03:24/14:26:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
27:03:24/14:26:32|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
27:03:24/14:26:32|TCP: Hash tables configured (established 4096 bind 4096)
27:03:24/14:26:32|TCP reno registered
27:03:24/14:26:32|NET: Registered protocol family 1
27:03:24/14:26:32|ROMFS MTD (C) 2007 Red Hat, Inc.
27:03:24/14:26:32|msgmni has been set to 214
27:03:24/14:26:32|io scheduler noop registered
27:03:24/14:26:32|io scheduler anticipatory registered
27:03:24/14:26:32|io scheduler deadline registered
27:03:24/14:26:32|io scheduler cfq registered (default)
27:03:24/14:26:32|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
27:03:24/14:26:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
27:03:24/14:26:32|console [ttyS0] enabled
27:03:24/14:26:32|brd: module loaded
27:03:24/14:26:32|loop: module loaded
27:03:24/14:26:32|Device Tree Probing 'ethernet'
27:03:24/14:26:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
27:03:24/14:26:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
27:03:24/14:26:32|XLlTemac: DCR address: 0x80
27:03:24/14:26:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
27:03:24/14:26:32|XLlTemac: Allocating DMA descriptors with kmalloc
27:03:24/14:26:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
27:03:24/14:26:33|XTemac: PHY detected at address 7.
27:03:24/14:26:33|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
27:03:24/14:26:33|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33|Using buffer write method
27:03:24/14:26:33|cfi_cmdset_0001: Erase suspend on write enabled
27:03:24/14:26:33|cmdlinepart partition parsing not available
27:03:24/14:26:33|RedBoot partition parsing not available
27:03:24/14:26:33|Creating 5 MTD partitions on "fc000000.flash":
27:03:24/14:26:33|0x000000000000-0x000000500000 : "golden_firmware"
27:03:24/14:26:33|0x000000500000-0x000000800000 : "golden_kernel"
... 1850 more lines ...
  556   Tue Apr 2 18:37:21 2024 TDS505 ADC offsets
S505 ADC offsets using pulser walkthrough data from data file R1

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )
Attachment 1: S505_calibration_data.txt
  $variables
  offset(   0) =    28.32
  offset(   1) =    17.71
  offset(   2) =    -9.75
  offset(   3) =    -5.24
  offset(   4) =   -10.84
  offset(   5) =     0.13
  offset(   6) =    28.59
  offset(   7) =    28.54
  offset(   8) =     0.83
  offset(   9) =   -12.15
  offset(  10) =    29.36
  offset(  11) =   -50.47
  offset(  12) =    10.03
  offset(  13) =   -26.90
  offset(  14) =    12.49
  offset(  15) =    10.57
  offset(  16) =     9.29
  offset(  17) =     1.09
  offset(  18) =    21.86
  offset(  19) =    -0.32
  offset(  20) =    22.59
  offset(  21) = -9999.99
  offset(  22) =   -20.22
  offset(  23) =     6.61
  offset(  24) =    -8.80
  offset(  25) =    -1.15
  offset(  26) =    13.08
  offset(  27) =    -9.03
  offset(  28) =    -2.67
  offset(  29) =    -9.42
  offset(  30) =     8.07
  offset(  31) =    19.01
  offset(  32) =     6.86
  offset(  33) =    15.16
  offset(  34) =    -3.89
  offset(  35) =   -18.80
  offset(  36) =    17.79
  offset(  37) =    16.45
  offset(  38) =    -0.74
  offset(  39) =     0.69
  offset(  40) =   -14.59
  offset(  41) =   -10.02
  offset(  42) =    11.38
  offset(  43) =    -2.57
  offset(  44) =    15.48
  offset(  45) =    14.57
  offset(  46) =     2.95
  offset(  47) =    30.70
  offset(  48) =   -12.55
  offset(  49) =   -15.61
  offset(  50) =    -6.61
  offset(  51) =    -7.10
  offset(  52) =   -33.20
  offset(  53) =    21.42
  offset(  54) =    -4.07
  offset(  55) =   -18.23
  offset(  56) =     8.83
  offset(  57) =   -22.74
  offset(  58) =    11.47
  offset(  59) =    10.67
  offset(  60) =   -28.65
  offset(  61) =    15.29
  offset(  62) =     5.76
  offset(  63) =   -11.57
  offset(  64) =     1.95
  offset(  65) = -9999.99
  offset(  66) =    22.52
  offset(  67) =    42.10
  offset(  68) =    -0.20
  offset(  69) =     9.78
  offset(  70) =   -11.48
  offset(  71) =    -0.03
  offset(  72) =   -12.50
  offset(  73) =    25.08
  offset(  74) =    12.77
  offset(  75) =    -2.76
  offset(  76) =     7.90
  offset(  77) =     9.85
  offset(  78) =     5.20
  offset(  79) =   -13.91
  offset(  80) =    18.82
  offset(  81) =    -8.25
  offset(  82) =    -8.40
  offset(  83) =    13.67
  offset(  84) =   -10.37
  offset(  85) =    28.32
  offset(  86) =    11.42
  offset(  87) =     2.74
  offset(  88) =   -24.88
  offset(  89) =    11.28
  offset(  90) =   -26.69
  offset(  91) =     0.91
  offset(  92) =     8.04
  offset(  93) =     7.64
  offset(  94) =    10.10
  offset(  95) =   -20.71
  offset(  96) =     6.72
  offset(  97) =   -12.63
  offset(  98) =    10.27
  offset(  99) =    16.05
  offset( 100) =     8.96
  offset( 101) =    33.57
  offset( 102) =    28.81
  offset( 103) =    23.49
  offset( 104) =     7.96
  offset( 105) =     6.07
  offset( 106) =    21.57
  offset( 107) =    13.06
  offset( 108) =     0.82
  offset( 109) =    -3.55
  offset( 110) =    -5.21
  offset( 111) =    11.48
  offset( 112) =    16.21
  offset( 113) =    -0.10
  offset( 114) =     3.87
  offset( 115) =    19.76
  offset( 116) =    -9.11
  offset( 117) =    24.64
  offset( 118) =     1.82
  offset( 119) = -9999.99
  offset( 120) =    22.80
  offset( 121) =    -7.37
  offset( 122) =   -18.32
  offset( 123) =   -24.74
  offset( 124) =    -4.71
  offset( 125) =    19.64
  offset( 126) =    53.61
  offset( 127) =     6.57
  offset( 128) =     3.21
  offset( 129) =     7.20
  offset( 130) =     5.55
  offset( 131) =    -3.40
  offset( 132) =   -16.40
  offset( 133) =    -1.71
  offset( 134) =   -14.85
  offset( 135) =   -29.53
  offset( 136) =     0.68
  offset( 137) =    -0.41
  offset( 138) =    17.59
  offset( 139) =   -28.92
  offset( 140) =    -8.79
  offset( 141) =    11.69
  offset( 142) =    -5.80
  offset( 143) =    -8.21
  offset( 144) =    13.90
  offset( 145) =    11.27
  offset( 146) =   -17.02
  offset( 147) =    -8.19
  offset( 148) =     6.14
  offset( 149) =   -13.13
  offset( 150) =    15.41
  offset( 151) =     0.12
  offset( 152) =    -5.02
  offset( 153) =     4.25
  offset( 154) =     3.84
  offset( 155) =    13.34
  offset( 156) =    16.79
  offset( 157) =   -36.21
  offset( 158) =    -2.18
  offset( 159) =    12.94
  offset( 160) =     2.05
  offset( 161) =   -17.31
  offset( 162) =    18.38
  offset( 163) =    -7.17
  offset( 164) =     6.59
  offset( 165) =   -13.36
  offset( 166) =   -12.11
  offset( 167) =    -3.91
  offset( 168) =    10.97
  offset( 169) =   -25.72
  offset( 170) =   -11.53
  offset( 171) =     1.94
  offset( 172) =   -12.76
  offset( 173) =    18.55
  offset( 174) =    13.49
  offset( 175) =    -9.45
  offset( 176) =   -10.19
  offset( 177) =    18.69
  offset( 178) =    24.49
  offset( 179) =    30.44
  offset( 180) =    41.95
  offset( 181) =     1.55
  offset( 182) =    20.88
  offset( 183) =     4.76
  offset( 184) =    15.80
  offset( 185) =    12.37
  offset( 186) =   -13.03
  offset( 187) =    15.64
  offset( 188) =    -8.44
  offset( 189) =    -3.02
  offset( 190) =    10.57
  offset( 191) =     6.88
  offset( 192) =   -17.81
  offset( 193) =    -5.77
  offset( 194) =   -18.29
  offset( 195) =    17.62
  offset( 196) =   -30.66
  offset( 197) =    -7.37
  offset( 198) =    36.32
  offset( 199) =     3.42
  offset( 200) =   -21.92
  offset( 201) =    15.88
  offset( 202) =   -17.33
  offset( 203) =    -2.71
  offset( 204) =   -21.06
  offset( 205) =    -4.53
  offset( 206) =    -1.59
  offset( 207) =    -9.54
  offset( 208) =    -0.04
  offset( 209) =    23.57
  offset( 210) =    13.83
  offset( 211) =    -9.28
  offset( 212) =    12.42
  offset( 213) =    23.85
  offset( 214) =     7.90
  offset( 215) =    -7.29
  offset( 216) =    26.18
  offset( 217) =    33.05
  offset( 218) =    32.10
  offset( 219) =    -8.10
  offset( 220) =    16.16
  offset( 221) =     3.94
  offset( 222) =   -18.22
  offset( 223) =   -16.84
  offset( 224) =     5.79
  offset( 225) =    21.07
  offset( 226) =    45.18
  offset( 227) =   -13.85
  offset( 228) =     6.76
  offset( 229) =    13.59
  offset( 230) =     8.28
  offset( 231) =    -9.32
  offset( 232) =    21.30
  offset( 233) =     7.02
  offset( 234) =    24.11
  offset( 235) =    16.26
  offset( 236) =    11.46
  offset( 237) =     2.16
  offset( 238) =     7.19
  offset( 239) =    18.45
  offset( 240) =     6.65
  offset( 241) =     6.56
  offset( 242) =   -33.59
  offset( 243) =    14.71
  offset( 244) =    21.28
  offset( 245) =    -0.73
  offset( 246) =   -10.29
  offset( 247) =    28.50
  offset( 248) =   -11.37
  offset( 249) =   -18.92
  offset( 250) =    23.12
  offset( 251) =     8.87
  offset( 252) =     9.64
  offset( 253) =    16.43
  offset( 254) =    -2.70
  offset( 255) =    26.14
  offset( 256) =    -4.99
  offset( 257) =   -10.88
  offset( 258) =    27.41
  offset( 259) =    -3.48
  offset( 260) =   -11.18
  offset( 261) =    12.60
  offset( 262) =    22.19
  offset( 263) =   -20.16
  offset( 264) =    20.32
  offset( 265) =     9.66
  offset( 266) =    13.30
  offset( 267) =   -12.64
  offset( 268) =    53.69
  offset( 269) =    23.15
  offset( 270) =    43.60
  offset( 271) =    48.58
  offset( 272) =   -13.12
  offset( 273) =   -12.57
  offset( 274) =    35.19
  offset( 275) =    14.84
  offset( 276) =    33.05
  offset( 277) =    13.73
  offset( 278) =    45.45
  offset( 279) =    55.49
  offset( 280) =    48.13
  offset( 281) =     3.86
  offset( 282) =    28.44
  offset( 283) =    23.73
  offset( 284) =    11.53
  offset( 285) =   -19.46
  offset( 286) =    10.81
  offset( 287) =    17.23
  offset( 288) =   -13.85
  offset( 289) =    30.17
  offset( 290) =    10.70
  offset( 291) =    28.06
  offset( 292) =   -14.35
  offset( 293) =    23.46
  offset( 294) =    15.16
  offset( 295) =   -25.15
  offset( 296) =   -15.03
  offset( 297) =   -24.74
  offset( 298) =    -5.94
... 215 more lines ...
  555   Tue Apr 2 12:36:25 2024 JB, CC, NHInstalling FEE64s of DSSSD2 cont.

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly

 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

 

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

 

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

 

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

 

- check test  and test - cable daisy chains are removed

 

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

 

  553   Thu Mar 28 09:18:53 2024 TDInstalling FEE64s of DSSSD2

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Quote:

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

AIDA - FEE Mapping
DSSSD 1 DSSSD2
FEE MAC FEE MAC
aida01 41:ba:8a aida06 41:05:15
aida02 41:f6:b7 aida07 41:f6:5a
aida03 41:d8:21 aida08 41:d7:cd
aida04 41:a0:71 aida10 41:d0:0E
aida05 41:cf:ac aida13 41:d8:2b
aida09 41:ee:10 aida14 42:0d:15
aida15 41:b4:0c aida11 41:EE:0f
aida12 41:ba:89 aida16 41:f6:ed

 

Going to try optimising noise now.

 

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

 

Firmware of aida11 updated from 0xea40704 to 0x3350706

 

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

 

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

 

  552   Wed Mar 27 14:22:35 2024 JB, NHInstalling FEE64s of DSSSD2

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

AIDA - FEE Mapping
DSSSD 1 DSSSD2
FEE MAC FEE MAC
aida01 41:ba:8a aida06 41:05:15
aida02 41:f6:b7 aida07 41:f6:5a
aida03 41:d8:21 aida08 41:d7:cd
aida04 41:a0:71 aida10 41:d0:0E
aida05 41:cf:ac aida13 41:d8:2b
aida09 41:ee:10 aida14 42:0d:15
aida15 41:b4:0c aida11 41:EE:0f
aida12 41:ba:89 aida16 41:f6:ed

 

Going to try optimising noise now.

 

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

 

Firmware of aida11 updated from 0xea40704 to 0x3350706

 

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

 

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

Attachment 1: Screenshot_2024-03-15_at_14-41-48_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-03-15_at_14-41-48_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2024-03-15_at_14-44-59_Statistics_aidas-gsi.png
Screenshot_2024-03-15_at_14-44-59_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2024-03-15_at_14-55-07_System_wide_Checks_aidas-gsi.png
Screenshot_2024-03-15_at_14-55-07_System_wide_Checks_aidas-gsi.png
Attachment 4: Screenshot_2024-03-15_at_14-57-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-03-15_at_14-57-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: Screenshot_from_2024-03-27_14-51-12.png
Screenshot_from_2024-03-27_14-51-12.png
Attachment 6: Screenshot_from_2024-03-27_14-53-05.png
Screenshot_from_2024-03-27_14-53-05.png
Attachment 7: Screenshot_from_2024-03-27_14-53-56.png
Screenshot_from_2024-03-27_14-53-56.png
Attachment 8: Screenshot_from_2024-03-27_15-31-10.png
Screenshot_from_2024-03-27_15-31-10.png
Attachment 9: Screenshot_from_2024-03-27_15-35-38.png
Screenshot_from_2024-03-27_15-35-38.png
Attachment 10: Screenshot_from_2024-03-27_15-36-13.png
Screenshot_from_2024-03-27_15-36-13.png
Attachment 11: Screenshot_from_2024-03-27_16-14-44.png
Screenshot_from_2024-03-27_16-14-44.png
Attachment 12: Screenshot_from_2024-03-27_16-15-00.png
Screenshot_from_2024-03-27_16-15-00.png
Attachment 13: Screenshot_from_2024-03-27_16-15-27.png
Screenshot_from_2024-03-27_16-15-27.png
Attachment 14: Screenshot_from_2024-03-27_16-15-56.png
Screenshot_from_2024-03-27_16-15-56.png
  551   Tue Mar 26 18:56:03 2024 TDUSB-controlled ac mains relay interlock box - wiring
Sensor ( 4 pins )

Red  +24V
Blue   0V
Yellow } contact 
Green  } closure in ( short these two wires together for logic 1 )

N.B. yellow and green wires of unused inputs must be connected together


Output ( 3 pins ) to USB-controlled ac mains relay

Red    NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue   NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM
  550   Tue Mar 26 10:32:05 2024 NH, JBTue 26 March
Taken 4 FEE64s from CRYRING (the 4 easiest to access)
  41:d8:2b
  41:f6:5a
  41:ee:71
  41:d0:0e

In addition a FEE without rails marked only 20 was found in my old office for 5
- This probably came in a shipment to/for repairs

DSSSD#1 unbiased

DESPEC Platform moved out of beam and gamma platform moved from AIDA
JB works on grounding the original 8 DSSSD#1 FEEs

Connected grounding from ribbon cable drain to adapter PCB
Ground loop connected

AIDA rates unchanged bad still fig 1
Waves fig 2 (p+n 6000-9000)
Waves fig 3 (n+n 8000-10000)
Statistics fig 4
Attachment 1: Screenshot_from_2024-03-26_14-39-35.png
Screenshot_from_2024-03-26_14-39-35.png
Attachment 2: Screenshot_from_2024-03-26_14-40-05.png
Screenshot_from_2024-03-26_14-40-05.png
Attachment 3: Screenshot_from_2024-03-26_14-40-30.png
Screenshot_from_2024-03-26_14-40-30.png
Attachment 4: Screenshot_from_2024-03-26_14-41-56.png
Screenshot_from_2024-03-26_14-41-56.png
Attachment 5: 20240326_144539.jpg
20240326_144539.jpg
Attachment 6: 20240326_144638.jpg
20240326_144638.jpg
Attachment 7: 20240326_144528.jpg
20240326_144528.jpg
Attachment 8: 20240326_144659.jpg
20240326_144659.jpg
Attachment 9: 20240326_144627.jpg
20240326_144627.jpg
  549   Fri Mar 22 08:31:39 2024 TDFriday 22 March
09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39
Attachment 1: Screenshot_from_2024-03-22_09-32-58.png
Screenshot_from_2024-03-22_09-32-58.png
Attachment 2: Screenshot_from_2024-03-22_09-34-54.png
Screenshot_from_2024-03-22_09-34-54.png
Attachment 3: Screenshot_from_2024-03-22_09-39-03.png
Screenshot_from_2024-03-22_09-39-03.png
Attachment 4: Screenshot_from_2024-03-22_09-40-10.png
Screenshot_from_2024-03-22_09-40-10.png
Attachment 5: Screenshot_from_2024-03-22_09-47-26.png
Screenshot_from_2024-03-22_09-47-26.png
Attachment 6: Screenshot_from_2024-03-22_09-49-33.png
Screenshot_from_2024-03-22_09-49-33.png
Attachment 7: Screenshot_from_2024-03-22_09-49-00.png
Screenshot_from_2024-03-22_09-49-00.png
Attachment 8: Screenshot_from_2024-03-22_09-48-25.png
Screenshot_from_2024-03-22_09-48-25.png
Attachment 9: Screenshot_from_2024-03-22_09-47-55.png
Screenshot_from_2024-03-22_09-47-55.png
Attachment 10: Screenshot_from_2024-03-22_10-00-47.png
Screenshot_from_2024-03-22_10-00-47.png
Attachment 11: Screenshot_from_2024-03-22_10-00-01.png
Screenshot_from_2024-03-22_10-00-01.png
Attachment 12: Screenshot_from_2024-03-22_09-58-56.png
Screenshot_from_2024-03-22_09-58-56.png
Attachment 13: Screenshot_from_2024-03-22_09-57-50.png
Screenshot_from_2024-03-22_09-57-50.png
Attachment 14: Screenshot_from_2024-03-22_09-56-45.png
Screenshot_from_2024-03-22_09-56-45.png
Attachment 15: Screenshot_from_2024-03-22_09-55-52.png
Screenshot_from_2024-03-22_09-55-52.png
Attachment 16: Screenshot_from_2024-03-22_09-54-18.png
Screenshot_from_2024-03-22_09-54-18.png
Attachment 17: Screenshot_from_2024-03-22_09-52-45.png
Screenshot_from_2024-03-22_09-52-45.png
Attachment 18: Screenshot_from_2024-03-22_09-52-13.png
Screenshot_from_2024-03-22_09-52-13.png
Attachment 19: Screenshot_from_2024-03-22_09-51-47.png
Screenshot_from_2024-03-22_09-51-47.png
Attachment 20: Screenshot_from_2024-03-22_10-04-22.png
Screenshot_from_2024-03-22_10-04-22.png
Attachment 21: Screenshot_from_2024-03-22_10-03-53.png
Screenshot_from_2024-03-22_10-03-53.png
Attachment 22: Screenshot_from_2024-03-22_14-29-18.png
Screenshot_from_2024-03-22_14-29-18.png
Attachment 23: Screenshot_from_2024-03-22_14-30-00.png
Screenshot_from_2024-03-22_14-30-00.png
Attachment 24: Screenshot_from_2024-03-22_14-31-19.png
Screenshot_from_2024-03-22_14-31-19.png
Attachment 25: Screenshot_from_2024-03-22_14-31-01.png
Screenshot_from_2024-03-22_14-31-01.png
Attachment 26: Screenshot_from_2024-03-22_14-31-57.png
Screenshot_from_2024-03-22_14-31-57.png
Attachment 27: Screenshot_from_2024-03-22_14-32-35.png
Screenshot_from_2024-03-22_14-32-35.png
Attachment 28: Screenshot_from_2024-03-22_14-40-04.png
Screenshot_from_2024-03-22_14-40-04.png
Attachment 29: Screenshot_from_2024-03-22_14-39-30.png
Screenshot_from_2024-03-22_14-39-30.png
Attachment 30: Screenshot_from_2024-03-22_14-38-22.png
Screenshot_from_2024-03-22_14-38-22.png
Attachment 31: Screenshot_from_2024-03-22_14-37-19.png
Screenshot_from_2024-03-22_14-37-19.png
Attachment 32: Screenshot_from_2024-03-22_14-36-28.png
Screenshot_from_2024-03-22_14-36-28.png
Attachment 33: Screenshot_from_2024-03-22_14-35-23.png
Screenshot_from_2024-03-22_14-35-23.png
Attachment 34: Screenshot_from_2024-03-22_14-34-38.png
Screenshot_from_2024-03-22_14-34-38.png
Attachment 35: Screenshot_from_2024-03-22_14-33-55.png
Screenshot_from_2024-03-22_14-33-55.png
Attachment 36: Screenshot_from_2024-03-22_14-41-21.png
Screenshot_from_2024-03-22_14-41-21.png
Attachment 37: Screenshot_from_2024-03-22_14-40-57.png
Screenshot_from_2024-03-22_14-40-57.png
Attachment 38: Capture_2.PNG
Capture_2.PNG
Attachment 39: Capture_3.PNG
Capture_3.PNG
  547   Fri Mar 22 08:29:55 2024 TDAnydesk restarted remotely
Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869
  546   Fri Mar 22 08:22:43 2024 NH, AM, MP, CCThu Mar 21
> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.
  545   Fri Mar 22 07:34:54 2024 NH, JBAu Beam
  544   Thu Mar 21 15:41:49 2024 NH, AM, MP, CCThu Mar 21
Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference
Attachment 1: Screenshot_from_2024-03-21_16-17-59.png
Screenshot_from_2024-03-21_16-17-59.png
Attachment 2: Screenshot_from_2024-03-21_16-18-25.png
Screenshot_from_2024-03-21_16-18-25.png
Attachment 3: Screenshot_from_2024-03-21_16-18-42.png
Screenshot_from_2024-03-21_16-18-42.png
Attachment 4: Screenshot_from_2024-03-21_16-39-12.png
Screenshot_from_2024-03-21_16-39-12.png
Attachment 5: Screenshot_from_2024-03-21_16-39-26.png
Screenshot_from_2024-03-21_16-39-26.png
Attachment 6: Screenshot_from_2024-03-21_16-39-39.png
Screenshot_from_2024-03-21_16-39-39.png
  543   Wed Mar 20 17:02:53 2024 NH/dev/sdd
The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)
  542   Wed Mar 20 12:22:27 2024 NHWed Mar 20

Turn on AIDA for Dry Run demonstrations and so on

All system wide checks, temp, bias OK

Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.

 Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing

 aida08 seems OK

Server running to MBS totally fine

18:00

Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation

AIDA is now powered off for the end of day

Attachment 1: Screenshot_from_2024-03-20_13-20-43.png
Screenshot_from_2024-03-20_13-20-43.png
Attachment 2: Screenshot_from_2024-03-20_13-20-54.png
Screenshot_from_2024-03-20_13-20-54.png
Attachment 3: Screenshot_from_2024-03-20_13-21-18.png
Screenshot_from_2024-03-20_13-21-18.png
Attachment 4: Screenshot_from_2024-03-20_13-21-38.png
Screenshot_from_2024-03-20_13-21-38.png
Attachment 5: Screenshot_from_2024-03-20_13-21-57.png
Screenshot_from_2024-03-20_13-21-57.png
Attachment 6: Screenshot_from_2024-03-20_13-22-12.png
Screenshot_from_2024-03-20_13-22-12.png
Attachment 7: Screenshot_from_2024-03-20_13-24-21.png
Screenshot_from_2024-03-20_13-24-21.png
Attachment 8: Screenshot_from_2024-03-20_13-25-02.png
Screenshot_from_2024-03-20_13-25-02.png
Attachment 9: Screenshot_from_2024-03-20_13-25-13.png
Screenshot_from_2024-03-20_13-25-13.png
  541   Tue Mar 19 10:17:30 2024 NHDry Run 2024 - 19th March 24
AIDA has 8 FEEs and 1 DSSSD 

Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
 
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
 (also on Grafana)

Temps OK fig2

System Checks (fig3-5)
 Clocks OK
 aida07 fails calibration, others OK
 WR OK (aida04 0x10, seems OK)
 FPGA OK

Did "Synchronise ASIC clocks" to align ASIC clocks

(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)

18:12 FRS is taking beam, AIDA is powered off and unbiased
Attachment 1: Screenshot_from_2024-03-19_13-44-34.png
Screenshot_from_2024-03-19_13-44-34.png
Attachment 2: Screenshot_from_2024-03-19_13-44-22.png
Screenshot_from_2024-03-19_13-44-22.png
Attachment 3: Screenshot_from_2024-03-19_13-44-04.png
Screenshot_from_2024-03-19_13-44-04.png
Attachment 4: Screenshot_from_2024-03-19_13-43-44.png
Screenshot_from_2024-03-19_13-43-44.png
Attachment 5: Screenshot_from_2024-03-19_13-43-24.png
Screenshot_from_2024-03-19_13-43-24.png
  540   Mon Mar 18 18:04:43 2024 NHPreparation for pre-s100 dry run (and test beam???)
In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again

AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08

This will allow the merger to run with 8 FEEs for 1 DSSD

dhcpd.conf updated

The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)

As should layout GSI_triple_test_renumber 

AFTER dry run:

Revert DHCP and prepare for full 16 FEEs 

Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders

Prepare a new Layout.mlf set
  539   Fri Mar 15 16:29:57 2024 NHLeakage currents
The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed

Minimum bias configuration:
 4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
 Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
 Drops can include 0 leakage current

Full adapter configuration:
 8 adapter boards, ground ring complete
 Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
 -60V again seems to be the turnover to a stable leakage current 

In both cases the leakage current during ramping appears basically the same as when settled

Full into FEEs
 8 adapter boards, fully connected to 8 FEEs
 The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
 Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
 This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
 During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)


I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages
  537   Thu Mar 14 13:00:23 2024 JB, NH, MA, AM, GAMounting and biasing DSSD 2

new Downstream DSSD2: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.1
20 3.8
30  
40  
50 4.515
60  
70 4.2
80 4.545
90 5.5
100 7.940
110  
120  

Voltage (V)

Current (uA)
10  
20  
30  
40  
50  
60  
70  
100 7.6
110 9.5
120 Breakdown
   
   

Wafer 3 (beam left) shows breakdown issues at 90V... curious

We replace with november beam triple: 3208-3/3208-21/3208-22

We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues

  536   Thu Mar 14 12:13:17 2024 JB, NH, MAMounting and biasing DSSD 1

Upstream bPlast mounted

new Upstream DSSD: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.345
20 3.6
30  
40  
50 4.5ish
60  
70 5.4
80  
90  
100 5.6
110  
120 5.7
   

V-I behaviour found to be nominal.

  535   Wed Mar 13 13:02:19 2024 JB, NH, HA, MADSSD 1 biasing tests

Dismounted Snout and biased DSSSD1 channels

leftmost waifer working

middle waifer reaches 90V then current ramps up

rightmost waifer reaches 80V then current ramps up

DSSSD2 had fingerprint near connections, as shown in the attached image some had been squashed.

Attachment 1: 20240313_142551.jpg
20240313_142551.jpg
  534   Tue Mar 12 16:16:49 2024 NH, TD, JB, HASummary of DSSSD Biasing 12.03
DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the   issue is internal to the snout

If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced

DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)

We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.

The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)

We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.

The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.

-

For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1

After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently

Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)

When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables
  533   Mon Mar 11 15:19:44 2024 JB, NH, Priyanka, Michael Armstrong, Helena AlbersTo Do: AIDA PCB tests

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

 - When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes.

- Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA.

2) Check that all ribbon cables are properly seated in the adaptor PCBs

- When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed.

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

- Done for the p+n side, can decide if this is needed for the n+n side.

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

-  Will have to decide further.

5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine).

- When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm.

- The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L.

6) Properly cover snout with black cloth and bias upstream.

- Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere.

Summary: from the p+n side test the voltage-current break down appears to come from inside the snout.

  532   Sun Mar 10 17:08:12 2024 NHAIDA FEE Layout + Cabling Plan for S100
Proposed FEE numbering and wiring plan for upcoming experiment S100 (2x Wide DSSSDs)

Image designed in draw.io, source attached 

FEE numbering is as S450, minimises cable movement from S505/Narrow AIDA
But means merger is not working with 1 DSSD (until all FEEs installed)

Wiring of adapter boards as from noise tests and what should work for DSSD bias
LK3 on middle bottom adapter to ground DSSD
LK1 on one n+n adapter to ground n+n side bias 
p+n has -ve voltage (w.r.t. ground) bias applied via lower adapter boards
ground loop grounds all adapter boards, except 2 p+n adapter boards which are grounded by the bias lemo shield instead

MACB layout also included, with expected NIM logic signals for the aida scalers:

1: Pulser/Sync clock (send to all subsystems, "trigger 3")0

3/4: Time Machine 
5/6: SC41L/R 

All other FEEs have their scaler available
(Scaler should be in left LEMO on MACB, right is output (AIDA->NIM/unused), bottom 4 are triggers from AIDA (unused)

Test circuit will not be used in experiment due to noise, but can be temporarily set up for pulser walkthrough

Revision 2 correct as of 27 March 2023
Attachment 1: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
Attachment 2: AIDA_S100_WiringPlan.drawio
<mxfile host="app.diagrams.net" modified="2024-03-09T18:35:12.515Z" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" etag="cLcX_EiphkJQIDoz3Eud" version="24.0.2" type="device">
  <diagram name="Page-1" id="rdp9_B-ufofv8BSNokWG">
    <mxGraphModel dx="1562" dy="846" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1920" pageHeight="1200" math="0" shadow="0">
      <root>
        <mxCell id="0" />
        <mxCell id="1" parent="0" />
        <mxCell id="Lok4d1QbW70blzeSAxq8-34" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-2" target="Lok4d1QbW70blzeSAxq8-3">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-2" value="9" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-35" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-3" target="Lok4d1QbW70blzeSAxq8-4">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-3" value="1" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-4" value="10" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-42" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-5" target="Lok4d1QbW70blzeSAxq8-6">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-5" value="13" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-43" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-6" target="Lok4d1QbW70blzeSAxq8-7">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-6" value="5" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-7" value="14" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-8" value="15" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-57" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-9" target="Lok4d1QbW70blzeSAxq8-8">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-9" value="7" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-40" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-15">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-54" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-51">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-56" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-9">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-10" value="16" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-44" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-11" target="Lok4d1QbW70blzeSAxq8-12">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-11" value="11" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-47" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-12" target="Lok4d1QbW70blzeSAxq8-13">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-12" value="3" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-32" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-14">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-55" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-49">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-13" value="12" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-61" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-14" target="Lok4d1QbW70blzeSAxq8-2">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-14" value="2" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="160" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-41" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-15" target="Lok4d1QbW70blzeSAxq8-5">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-15" value="6" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="80" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-37" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-16" target="Lok4d1QbW70blzeSAxq8-10">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-16" value="8" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="880" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-17" value="4" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="800" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-18" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="514" y="416" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-21" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="514" y="496" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-36" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-22" target="Lok4d1QbW70blzeSAxq8-13">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-22" value="&lt;div&gt;LK1&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="790" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-23" value="LK1" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="870" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-49" value="&lt;div&gt;HV CH0&lt;br&gt;NEGATIVE&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="990" y="450" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-51" value="&lt;div&gt;HV CH1&lt;br&gt;NEGATIVE&lt;br&gt;&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="990" y="530" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-59" value="" style="shape=sumEllipse;perimeter=ellipsePerimeter;whiteSpace=wrap;html=1;backgroundOutline=1;" vertex="1" parent="1">
          <mxGeometry x="484" y="305" width="30" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-60" value="Beam Into Page" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="530" y="305" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-62" value="LK1 = Bias -&amp;gt; GND&lt;br&gt;LK3 = DSSD GND -&amp;gt; GND" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="770" y="125" width="170" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-89" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="230" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-65" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-66" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-67" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-70" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-71" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-72" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-77" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-78" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-79" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-80" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-81" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-84" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-85" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-86" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-87" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-88" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-90" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="350" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-91" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-92" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-93" value="1" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-94" value="2" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-95" value="3" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-96" value="4" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-97" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-98" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-99" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-100" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-101" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-102" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-103" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-104" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-105" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-106" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-107" value="5" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="470" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-108" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-109" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-110" value="5" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-111" value="6" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-112" value="7" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-113" value="8" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-114" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-115" value="6" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-116" value="7" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-117" value="8" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-118" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-119" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-120" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-121" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-122" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-123" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-124" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="590" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-125" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-126" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-127" value="9" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-128" value="10" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-129" value="11" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
... 196 more lines ...
Attachment 3: AIDA_S100_WiringPlan(1).drawio
<mxfile host="app.diagrams.net" modified="2024-03-27T12:12:06.799Z" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" etag="0KC3XQLCIUXvtcbPW53b" version="23.1.1" type="device">
  <diagram name="Page-1" id="rdp9_B-ufofv8BSNokWG">
    <mxGraphModel dx="794" dy="1153" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1920" pageHeight="1200" math="0" shadow="0">
      <root>
        <mxCell id="0" />
        <mxCell id="1" parent="0" />
        <mxCell id="Lok4d1QbW70blzeSAxq8-34" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-2" target="Lok4d1QbW70blzeSAxq8-3" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-2" value="9" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-35" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-3" target="Lok4d1QbW70blzeSAxq8-4" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-3" value="1" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-4" value="5" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-42" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-5" target="Lok4d1QbW70blzeSAxq8-6" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-5" value="10" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-43" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-6" target="Lok4d1QbW70blzeSAxq8-7" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-6" value="14" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-7" value="13" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-8" value="11" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-57" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" parent="1" source="Lok4d1QbW70blzeSAxq8-9" target="Lok4d1QbW70blzeSAxq8-8" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-9" value="7" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-40" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-15" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-54" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-51" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-56" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-9" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-10" value="16" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-44" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-11" target="Lok4d1QbW70blzeSAxq8-12" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-11" value="15" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-47" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-12" target="Lok4d1QbW70blzeSAxq8-13" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-12" value="3" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-32" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-14" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-55" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-49" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-13" value="12" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-61" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-14" target="Lok4d1QbW70blzeSAxq8-2" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-14" value="2" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="160" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-41" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-15" target="Lok4d1QbW70blzeSAxq8-5" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-15" value="6" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="80" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-37" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-16" target="Lok4d1QbW70blzeSAxq8-10" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-16" value="8" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="880" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-17" value="4" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="800" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-18" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="514" y="416" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-21" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="514" y="496" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-36" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-22" target="Lok4d1QbW70blzeSAxq8-13" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-22" value="&lt;div&gt;LK1&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="790" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-23" value="LK1" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="870" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-49" value="&lt;div&gt;HV CH0&lt;br&gt;NEGATIVE&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="990" y="450" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-51" value="&lt;div&gt;HV CH1&lt;br&gt;NEGATIVE&lt;br&gt;&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="990" y="530" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-59" value="" style="shape=sumEllipse;perimeter=ellipsePerimeter;whiteSpace=wrap;html=1;backgroundOutline=1;" parent="1" vertex="1">
          <mxGeometry x="484" y="305" width="30" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-60" value="Beam Into Page" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="530" y="305" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-62" value="LK1 = Bias -&amp;gt; GND&lt;br&gt;LK3 = DSSD PCB GND -&amp;gt; GND" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="760" y="125" width="180" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-89" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="230" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-65" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-66" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-67" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-70" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-71" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-72" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-77" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-78" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-79" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-80" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-81" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-84" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-85" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-86" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-87" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-88" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-90" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="350" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-91" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-92" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-93" value="1" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-94" value="2" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-95" value="3" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-96" value="4" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-97" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-98" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-99" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-100" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-101" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-102" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-103" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-104" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-105" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-106" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-107" value="5" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="470" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-108" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-109" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-110" value="5" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-111" value="6" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-112" value="7" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-113" value="8" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-114" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-115" value="6" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-116" value="7" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-117" value="8" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-118" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-119" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-120" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-121" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-122" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-123" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-124" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="590" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-125" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-126" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-127" value="9" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-128" value="10" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-129" value="11" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
... 207 more lines ...
Attachment 4: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
  531   Fri Mar 8 16:05:57 2024 JB, CC, TD, NHFriday 8 March

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Bias Voltage (V) Current (uA)
+10 2.150
+20 3.300
+30 4.035
+40 4.550
+50 4.955
+60 5.255
+70 5.490
+80 5.650
+90 5.730
+100 5.780
+110 5.825
+120 5.860

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

 

Attachment 1: IV_test_AIDA.xlsx
Attachment 2: chart.png
chart.png
Attachment 3: Downstream_positive_bias_vs._Current_(uA).png
Downstream_positive_bias_vs._Current_(uA).png
  530   Thu Mar 7 15:15:29 2024 JB, NH, CCAIDA HV Bias Test IV Curve
AIDA I-V Test
Voltage (V) Current (uA)
10 1.43
20 2.065
30 2.415
40 2.64
50 2.825
60 2.99
70 3.185
80 3.58
90 5.01
100 9.4

- Voltage-Current test of newly installed AIDA snout. Breakdown observed at around ~92 V. Probably caused my light leakage into the snout, will investigate further.

- Tried to cover the snout with a black cloth. This did not change the breakdown behaviour.

  529   Thu Mar 7 11:09:05 2024 HA, JB, CC, TD, MGThursday 7 March
Snout assembly

DSSSDs

Upstream   3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12

p+n junction side bias continuity checked OK for *all* wafers
 all c. 22 Ohm as expected

n+n Ohmic side bias continuity checked OK for both DSSSDs
 upstream 73 Ohm, downstream 92 Ohm

n+n Ohmic side bias wafer #2 
 positioned at top of snout assembly ( lower stage snout side marked 'T' )

p+n junction side faces downstream


Distances

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm

Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side 


13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.

Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.

We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and 
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.

Distances re-measured as

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm
  528   Tue Mar 5 09:10:01 2024 MA, JB, TDTuesday 5 March
Old ribbon cable assemblies - attachments 1  2

Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors

p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm

Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm



Manufacture of new ribbon cable assemblies - attachment 3

8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables


17.52 from Helena

"These are the detector positions within the snout:

Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame

Upstream AIDA @ 513 mm

Downstream AIDA @ 523 mm

This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is 
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by 
10mm.

The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position."
Attachment 1: 20240305_101140.jpg
20240305_101140.jpg
Attachment 2: 20240305_101146.jpg
20240305_101146.jpg
Attachment 3: 20240305_165949.jpg
20240305_165949.jpg
  527   Mon Mar 4 11:35:30 2024 TDMonday 4 March
12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida08

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps - attachment 1
       aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
       aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch

       aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 13 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11
Attachment 1: Screenshot_from_2024-03-04_12-27-04.png
Screenshot_from_2024-03-04_12-27-04.png
Attachment 2: Screenshot_from_2024-03-04_12-37-24.png
Screenshot_from_2024-03-04_12-37-24.png
Attachment 3: Screenshot_from_2024-03-04_12-27-34.png
Screenshot_from_2024-03-04_12-27-34.png
Attachment 4: Screenshot_from_2024-03-03_19-02-33.png
Screenshot_from_2024-03-03_19-02-33.png
Attachment 5: Screenshot_from_2024-03-04_12-29-44.png
Screenshot_from_2024-03-04_12-29-44.png
Attachment 6: Screenshot_from_2024-03-04_12-33-43.png
Screenshot_from_2024-03-04_12-33-43.png
Attachment 7: Screenshot_from_2024-03-04_12-32-50.png
Screenshot_from_2024-03-04_12-32-50.png
Attachment 8: Screenshot_from_2024-03-04_12-32-11.png
Screenshot_from_2024-03-04_12-32-11.png
Attachment 9: Screenshot_from_2024-03-04_12-31-30.png
Screenshot_from_2024-03-04_12-31-30.png
Attachment 10: Screenshot_from_2024-03-04_12-34-49.png
Screenshot_from_2024-03-04_12-34-49.png
Attachment 11: Screenshot_from_2024-03-04_12-34-40.png
Screenshot_from_2024-03-04_12-34-40.png
  526   Sun Mar 3 16:06:11 2024 TDSunday 3 March
17.00 FEE64 41:a0:71 ASIC #2 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida04

      Unable to boot nnrpi2 - manual/local control of BNC PB-5 and FEE64 PSU - no interlock

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps OK - attachment 1

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 15 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11

      MIDAS configuration - attachment 12

      DHCP config - attachment 13



18.30 FEE64 41:f6:b7 ASIC #1 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida02

      All system wide checks OK

      FEE64 temps - attachment 14
       aida02 ASIC temperature c. 512 deg C !
       PSU/Virtex temps OK
       Mezzanine ambient to touch - assume faulty sensor/poor connection?

      aida02 ASIC settings - attachment 15

      WR timestamp OK - attachment 16

      ADC data item stats OK - attachment 4

      aida04 Rate spectrum - attachment 18

      aida02 1.8.W spectra - 20us FSR - attachment 19-20

      aida02 *.*.L spectra - attachments 21-24
       1.8.L pulser peak width 16 ch FWHM


19.30 Power OFF



     
Attachment 1: Screenshot_from_2024-03-03_16-54-42.png
Screenshot_from_2024-03-03_16-54-42.png
Attachment 2: Screenshot_from_2024-03-03_16-55-14.png
Screenshot_from_2024-03-03_16-55-14.png
Attachment 3: Screenshot_from_2024-03-03_16-55-21.png
Screenshot_from_2024-03-03_16-55-21.png
Attachment 4: Screenshot_from_2024-03-03_17-07-26.png
Screenshot_from_2024-03-03_17-07-26.png
Attachment 5: Screenshot_from_2024-03-03_16-53-52.png
Screenshot_from_2024-03-03_16-53-52.png
Attachment 6: Screenshot_from_2024-03-03_17-02-09.png
Screenshot_from_2024-03-03_17-02-09.png
Attachment 7: Screenshot_from_2024-03-03_17-01-52.png
Screenshot_from_2024-03-03_17-01-52.png
Attachment 8: Screenshot_from_2024-03-03_17-00-18.png
Screenshot_from_2024-03-03_17-00-18.png
Attachment 9: Screenshot_from_2024-03-03_16-59-23.png
Screenshot_from_2024-03-03_16-59-23.png
Attachment 10: Screenshot_from_2024-03-03_16-58-43.png
Screenshot_from_2024-03-03_16-58-43.png
Attachment 11: Screenshot_from_2024-03-03_16-57-09.png
Screenshot_from_2024-03-03_16-57-09.png
Attachment 12: startup.tcl

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
     set ACQSERVERS [list aida01 aida03 aida04 aida05 aida08]
     variable ACQSERVER [first $ACQSERVERS]
     variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


Attachment 13: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host edinburgh {
      hardware ethernet 00:14:4f:9b:6e:7a;
      fixed-address 192.168.11.118;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:a0:71;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }


   host aida05 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:42:0d:15;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   #host aida06 {
   #  hardware ethernet d8:80:39:41:ee:72;
   #  fixed-address 192.168.11.6;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   #}

   #host aida07 {
   #  hardware ethernet d8:80:39:41:b4:0c;
   #  fixed-address 192.168.11.7;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   #}

   #host aida08 {
   #  hardware ethernet d8:80:39:41:ba:2b;
   #  fixed-address 192.168.11.8;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   #}

   #host aida13 {
   #  hardware ethernet d8:80:39:42:d:15;
   #  fixed-address 192.168.11.13;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   #}

#   host aida14 {
#     hardware ethernet d8:80:39:42:d:b;
#     fixed-address 192.168.11.14;
#    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
#   }

   #host aida05 {
   #  hardware ethernet d8:80:39:41:ee:10;
   #  fixed-address 192.168.11.5;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   #}

   #host aida14 {
   #  hardware ethernet d8:80:39:41:f6:ed;
   #  fixed-address 192.168.11.14;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   #}
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;







}


Attachment 14: Screenshot_from_2024-03-03_18-53-03.png
Screenshot_from_2024-03-03_18-53-03.png
Attachment 15: Screenshot_from_2024-03-03_18-53-16.png
Screenshot_from_2024-03-03_18-53-16.png
Attachment 16: Screenshot_from_2024-03-03_18-53-41.png
Screenshot_from_2024-03-03_18-53-41.png
Attachment 17: Screenshot_from_2024-03-03_19-02-33.png
Screenshot_from_2024-03-03_19-02-33.png
Attachment 18: Screenshot_from_2024-03-03_18-53-47.png
Screenshot_from_2024-03-03_18-53-47.png
Attachment 19: Screenshot_from_2024-03-03_19-01-41.png
Screenshot_from_2024-03-03_19-01-41.png
Attachment 20: Screenshot_from_2024-03-03_19-01-31.png
Screenshot_from_2024-03-03_19-01-31.png
Attachment 21: Screenshot_from_2024-03-03_19-00-11.png
Screenshot_from_2024-03-03_19-00-11.png
Attachment 22: Screenshot_from_2024-03-03_18-59-24.png
Screenshot_from_2024-03-03_18-59-24.png
Attachment 23: Screenshot_from_2024-03-03_18-58-39.png
Screenshot_from_2024-03-03_18-58-39.png
Attachment 24: Screenshot_from_2024-03-03_18-57-57.png
Screenshot_from_2024-03-03_18-57-57.png
  525   Mon Feb 26 13:17:51 2024 TDOffline analysis of DEC23/R9_6
First pass analysis of data file /TapeData/DEC23/R9_6

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  234532726 (   99567.1 Hz)
        Other data format:   27387282 (   11626.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         53 (       0.0 Hz)
                               RESUME:         53 (       0.0 Hz)
                              SYNC100:      40775 (      17.3 Hz)
                              WR48-63:      40775 (      17.3 Hz)
                           FEE64 disc:    2928025 (    1243.0 Hz)
                             MBS info:   24377601 (   10349.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     287073 (     121.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2355.524 s

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   59013868     372005          0          0         20         20       9888       9888     281563      70626          0      55467
  1   31445079    1098714          0          0          2          2       5017       5017     243009     845667          0      38876
  2   22895499   11841669          0          0          1          1       5294       5294     482655   11348424          0      42748
  3   18228498   11705354          0          0          0          0       3913       3913     416619   11280909          0      78526
  4   25500772    1237178          0          0          3          3       4320       4320     396557     831975          0      28918
  5          4          0          0          0          0          0          0          0          0          0          0          0
  6   15933921     394210          0          0          0          0       2560       2560     389090          0          0      11489
  7   61515085     738152          0          0         27         27       9783       9783     718532          0          0      31049

FEE64 configuration https://elog.ph.ed.ac.uk/AIDA/935


Note

- aida06 not connected to DSSSD ( cabling broken  - will be replaced later this week )

- LEC ( 20MeV FSR ) data ADC offset corrected 
   ADC offsets available for 489 of 512 channels and are included in the analysis - 23 strips for which no ADC offsets could be calculated ( usually because there was 
no 
pulser data ) are not included in analysis 
- LEC ( 20MeV FSR ) front-back strip energy difference cut +/- 50 channels ( c. +/- 280keV )
- HEC ( 2GeV FSR ) front-back strip energy difference cut +/-200 channels ( c. +/- 1120MeV )

- LEC ADC data: 13 < channel < 188 ( c. 73-1053keV )
- HEC ADC data: > 13 channels ( c. 73MeV )
- FEE64 hardware thresholds: LEC c. 100keV, HEC c. 200MeV 




Attachments 1 & 2 - per pixel HEC-LEC event time difference spectra - 4.096us/channel

Attachments 3 & 4 - per pixel HEC-LEC event time difference spectra - 65.536us/channel

Attachment 5 - DSSSD x-y hit pattern: HEC-LEC event time difference < 4.3s
 Majority of events associated with 'hot' p+n junction strips - few plausible decay candidate events - as expected.
 z-scale - semi-logarithmic - scattered events are single counts ( blue )
 1 count => rate ~ 1/2355s ~ 0.0004Hz, the great majority of pixels have zero counts

Attachment 6 - LEC ( decay ) and HEC ( implant ) events - 262.144us/channel ( 65536 channels = 17.2s ) 

HEC implants channels 12000-25000 = 216 counts => HEC rate ( in spill ) 63.4Hz
LEC decays channels 12000-25000 = 3865 counts => LEC rate ( in spill ) 1134.1Hz
LEC decays channels 25000-38000 = 2790 counts => LEC rate ( inter spill ) 818.7Hz

Attachment 7 - per pixel HEC-HEC event time difference spectra - 4.096us/channel

Attachment 8 - per pixel LEC-LEC event time difference spectra - 4.096us/channel

Attachment 9 - variables.dat
This is a ( Fortran ) Namelist I/O data file containing of the ADC offsets, FEE64 configuration, LEC and HEC energy difference windows 

The ADC offset channel number is calculated as

channel = channel_ident + ( module * 64 ) where module = 0-7 corresponding to AIDA FEE64s aida01-aida08

and is used as follows

ADC data = INT ( RSHIFT( ABS( 32768 - data( i ) ), 3 ) - offset( i ) + 0.5 )

where data(i) is the ADC data item for channel i, offset(i) is the ADC offset for channel i

An ADC offset of -9999 means there was no pulser data for this channel in data files R5 and R11.

               
Attachment 1: Screenshot_from_2024-03-01_11-41-11.png
Screenshot_from_2024-03-01_11-41-11.png
Attachment 2: Screenshot_from_2024-03-01_11-41-30.png
Screenshot_from_2024-03-01_11-41-30.png
Attachment 3: Screenshot_from_2024-03-01_11-42-06.png
Screenshot_from_2024-03-01_11-42-06.png
Attachment 4: Screenshot_from_2024-03-01_11-42-22.png
Screenshot_from_2024-03-01_11-42-22.png
Attachment 5: Screenshot_from_2024-03-01_11-43-13.png
Screenshot_from_2024-03-01_11-43-13.png
Attachment 6: Screenshot_from_2024-03-01_11-44-33.png
Screenshot_from_2024-03-01_11-44-33.png
Attachment 7: Screenshot_from_2024-03-01_11-38-52.png
Screenshot_from_2024-03-01_11-38-52.png
Attachment 8: Screenshot_from_2024-03-01_11-40-37.png
Screenshot_from_2024-03-01_11-40-37.png
Attachment 9: variables.dat
  $variables
  offset(   0) =    10.71
  offset(   1) =    14.56
  offset(   2) =   -13.88
  offset(   3) =    -9.85
  offset(   4) =   -18.84
  offset(   5) =    -3.41
  offset(   6) =    24.00
  offset(   7) =    23.10
  offset(   8) =    -2.39
  offset(   9) =   -15.90
  offset(  10) =    25.89
  offset(  11) =   -55.27
  offset(  12) =     6.39
  offset(  13) =   -31.83
  offset(  14) =     7.94
  offset(  15) =     7.69
  offset(  16) =     3.46
  offset(  17) =    -3.12
  offset(  18) =    16.24
  offset(  19) =    -6.69
  offset(  20) =    18.42
  offset(  21) = -9999.99
  offset(  22) =   -22.39
  offset(  23) =     1.94
  offset(  24) =   -11.29
  offset(  25) =    -5.55
  offset(  26) =     7.88
  offset(  27) =    -4.32
  offset(  28) =    -7.79
  offset(  29) =   -13.60
  offset(  30) =     2.61
  offset(  31) =    14.95
  offset(  32) =     4.43
  offset(  33) = -9999.99
  offset(  34) =    -6.61
  offset(  35) =   -22.58
  offset(  36) = -9999.99
  offset(  37) =    11.90
  offset(  38) = -9999.99
  offset(  39) =    -1.80
  offset(  40) =   -26.27
  offset(  41) = -9999.99
  offset(  42) =    -1.99
  offset(  43) =    -3.66
  offset(  44) = -9999.99
  offset(  45) =     7.10
  offset(  46) =    -0.31
  offset(  47) =    26.80
  offset(  48) =   -16.60
  offset(  49) = -9999.99
  offset(  50) =    -9.08
  offset(  51) =   -10.53
  offset(  52) =   -28.24
  offset(  53) =    13.09
  offset(  54) =   -12.42
  offset(  55) =   -22.52
  offset(  56) =     3.27
  offset(  57) =   -26.47
  offset(  58) =     7.17
  offset(  59) =     5.31
  offset(  60) =   -29.79
  offset(  61) =    19.75
  offset(  62) =     2.12
  offset(  63) =   -22.37
  offset(  64) =     3.02
  offset(  65) =     1.74
  offset(  66) =    23.39
  offset(  67) =    43.87
  offset(  68) =   -15.18
  offset(  69) =    11.78
  offset(  70) =   -14.59
  offset(  71) =     1.89
  offset(  72) =   -15.75
  offset(  73) =    25.22
  offset(  74) =     7.83
  offset(  75) = -9999.99
  offset(  76) =     6.46
  offset(  77) =    12.32
  offset(  78) =     7.61
  offset(  79) =   -11.58
  offset(  80) =    14.92
  offset(  81) =    -5.12
  offset(  82) = -9999.99
  offset(  83) =    13.14
  offset(  84) =   -13.39
  offset(  85) =    29.08
  offset(  86) =     9.65
  offset(  87) =     5.07
  offset(  88) =   -29.50
  offset(  89) = -9999.99
  offset(  90) =   -32.90
  offset(  91) =    -1.58
  offset(  92) =    10.27
  offset(  93) =    10.60
  offset(  94) =     6.11
  offset(  95) =   -17.37
  offset(  96) = -9999.99
  offset(  97) =   -15.29
  offset(  98) =     5.60
  offset(  99) =    18.18
  offset( 100) =     6.95
  offset( 101) =    34.63
  offset( 102) =    24.44
  offset( 103) = -9999.99
  offset( 104) =    10.31
  offset( 105) =     2.06
  offset( 106) =    24.74
  offset( 107) =    14.27
  offset( 108) =    -2.56
  offset( 109) =    -3.93
  offset( 110) = -9999.99
  offset( 111) =     8.74
  offset( 112) =    10.62
  offset( 113) =     8.13
  offset( 114) =    -9.11
  offset( 115) =    28.90
  offset( 116) =   -22.43
  offset( 117) =    15.39
  offset( 118) =     8.35
  offset( 119) = -9999.99
  offset( 120) =    23.88
  offset( 121) =    -8.30
  offset( 122) = -9999.99
  offset( 123) =   -22.34
  offset( 124) =    -3.13
  offset( 125) = -9999.99
  offset( 126) = -9999.99
  offset( 127) = -9999.99
  offset( 128) =    -5.53
  offset( 129) =    13.01
  offset( 130) =     4.93
  offset( 131) =    -6.37
  offset( 132) =   -14.43
  offset( 133) =    -1.17
  offset( 134) =   -13.50
  offset( 135) =   -29.29
  offset( 136) =     0.38
  offset( 137) =    -2.31
  offset( 138) =    17.01
  offset( 139) =   -28.94
  offset( 140) =    -9.94
  offset( 141) =    10.87
  offset( 142) =    -6.41
  offset( 143) =   -10.41
  offset( 144) =    13.21
  offset( 145) =    10.55
  offset( 146) =   -16.63
  offset( 147) =    -9.88
  offset( 148) =     3.68
  offset( 149) =   -13.44
  offset( 150) =    16.01
  offset( 151) =    -0.66
  offset( 152) =    -4.78
  offset( 153) =     3.88
  offset( 154) =     1.75
  offset( 155) =    14.87
  offset( 156) =    14.93
  offset( 157) = -9999.99
  offset( 158) =    -1.27
  offset( 159) =    11.46
  offset( 160) =     1.60
  offset( 161) =   -17.07
  offset( 162) = -9999.99
  offset( 163) =    -7.67
  offset( 164) =     6.72
  offset( 165) =   -14.06
  offset( 166) =   -13.29
  offset( 167) =    -5.26
  offset( 168) =     9.71
  offset( 169) =   -26.00
  offset( 170) =   -13.22
  offset( 171) =     1.65
  offset( 172) =   -16.16
  offset( 173) =    16.84
  offset( 174) =    10.04
  offset( 175) =   -13.46
  offset( 176) =   -11.55
  offset( 177) =    18.48
  offset( 178) =    26.41
  offset( 179) =    26.68
  offset( 180) =    52.12
  offset( 181) =    -4.79
  offset( 182) =    15.28
  offset( 183) =     7.10
  offset( 184) =    14.26
  offset( 185) =    12.19
  offset( 186) =   -11.96
  offset( 187) =    12.95
  offset( 188) =   -14.44
  offset( 189) =    -0.90
  offset( 190) =     8.53
  offset( 191) =     5.83
  offset( 192) =    14.00
  offset( 193) =    23.65
  offset( 194) =    45.50
  offset( 195) =    19.86
  offset( 196) =    20.01
  offset( 197) =    -5.01
  offset( 198) =    10.92
  offset( 199) =    10.42
  offset( 200) =   -23.83
  offset( 201) =    -8.06
  offset( 202) =    -1.51
  offset( 203) =    11.88
  offset( 204) =   -17.23
  offset( 205) =    22.72
  offset( 206) =   -15.70
  offset( 207) =    69.14
  offset( 208) =   -31.45
  offset( 209) =     4.00
  offset( 210) =     3.27
  offset( 211) =    -1.41
  offset( 212) =    17.72
  offset( 213) =    -9.82
  offset( 214) =    42.36
  offset( 215) =   -14.39
  offset( 216) =   -10.26
  offset( 217) =    16.12
  offset( 218) =   -13.05
  offset( 219) =    32.84
  offset( 220) =     8.33
  offset( 221) =   -31.72
  offset( 222) =    -6.97
  offset( 223) =     9.41
  offset( 224) =    13.02
  offset( 225) =    -4.24
  offset( 226) =     2.84
  offset( 227) =    19.86
  offset( 228) =     1.54
  offset( 229) =    -1.36
  offset( 230) =    -4.71
  offset( 231) =    -8.29
  offset( 232) =    17.26
  offset( 233) =   -27.21
  offset( 234) =     0.91
  offset( 235) =     0.88
  offset( 236) =    25.52
  offset( 237) =    -5.75
  offset( 238) =   -14.34
  offset( 239) =    -1.61
  offset( 240) =     3.03
  offset( 241) =    16.54
  offset( 242) =    -5.54
  offset( 243) =   -18.73
  offset( 244) =     9.95
  offset( 245) =     7.49
  offset( 246) =     6.72
  offset( 247) =   -10.82
  offset( 248) =    22.57
  offset( 249) =    25.84
  offset( 250) =    24.76
  offset( 251) =     9.43
  offset( 252) =    -5.13
  offset( 253) =     6.02
  offset( 254) =     1.75
  offset( 255) =    38.44
  offset( 256) = -9999.99
  offset( 257) =     4.87
  offset( 258) = -9999.99
  offset( 259) =   -22.56
  offset( 260) =     8.35
  offset( 261) =     3.63
  offset( 262) =   -21.39
  offset( 263) =    17.77
  offset( 264) =    -5.50
  offset( 265) =    -3.87
  offset( 266) =     1.14
  offset( 267) =    -1.08
  offset( 268) =     9.44
  offset( 269) =     1.99
  offset( 270) =    -5.29
  offset( 271) =    12.22
  offset( 272) =     4.93
  offset( 273) =    -0.64
  offset( 274) =    22.92
  offset( 275) =   -18.21
  offset( 276) =     0.20
  offset( 277) =    -6.97
  offset( 278) =    14.96
  offset( 279) =    14.56
  offset( 280) =    14.59
  offset( 281) =    -5.61
  offset( 282) =    15.35
  offset( 283) =   -22.48
  offset( 284) =     9.83
  offset( 285) =    19.83
  offset( 286) =   -20.28
  offset( 287) =    27.19
  offset( 288) =    11.79
  offset( 289) =    -8.16
  offset( 290) =    81.33
  offset( 291) =    -4.18
  offset( 292) =   -12.11
  offset( 293) =    -1.70
  offset( 294) =   -12.91
  offset( 295) =     1.57
  offset( 296) =   -12.75
  offset( 297) =   -23.81
  offset( 298) =     4.09
... 227 more lines ...
  524   Wed Jan 31 15:27:07 2024 TDPhotos of water flow & dew point sensor interlock box
Attachment 1 - 4 socket input connectors ( from sensor to interlock box )

Attachment 2 & 2 - 24V dc relays and internal wiring

Attachment 4 - 3 socket input connector ( from interlock box to USB-controlled ac mains relay )
Attachment 1: 20240131_160205.jpg
20240131_160205.jpg
Attachment 2: 20240131_145249.jpg
20240131_145249.jpg
Attachment 3: 20240131_145340.jpg
20240131_145340.jpg
Attachment 4: 20240131_160341.jpg
20240131_160341.jpg
  523   Mon Nov 13 11:46:54 2023 TDSummary of AIDA 24cm x 8cm 'triple' DSSSD tests
Attachment 1: Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf
Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf
Attachment 2: Summary_of_AIDA_tests_2022-3_-_Dec_2023.pptx
  522   Thu Oct 12 14:01:43 2023 TDMSL type BB18 24cm x 8cm DSSSD test - update
DSSSD MSL type BB18 24cm x 8cm 3208-3/3208-21/3208-22

FEE64 configuration see https://elog.ph.ed.ac.uk/AIDA/872 attachment 2

Bias -150V -6.590uA ambient temperature +24.7 deg C d.p. +13.7 deg C RH 50.3%

BNC PB-5
amplitude 10.0V
attenuation x10
decay time 1ms
tail pulse
frequency 25Hz

PB-5 output direct to p+n junction side FEE64 aida01 or aida12, or n+n Ohmic side FEE64 aida02


aida01 1.8.L pulser peak width  61 ch FWHM ~  46keV FWHM => 5s threshold 98keV 
aida12 1.8.L pulser peak width  56 ch FWHM ~  42keV FWHM => 5s threshold 89keV
aida02 1.8.L pulser peak width 102 ch FWHM ~  77keV FWHM => 5s threshold 163keV


slow comparator  0xa 100keV ( all p+n junction FEE64s ) 
                 0xf 150keV ( all n+n Ohmic FEE64s )

per FEE64 Rate spectra - attachment 1
 p+n FEE64s ( aida010, aida01, aida09, aida12, aida03, aida11 ) rates dominated by hot channels, other channels typically <1Hz ( 25Hz pulser to aida12 )
 n+n FEE64s ( aida02, aida04 ) rates ~ 10-20Hz/channel
 Note aida06 and aida08 are not connected to anything and should be ignored


ADC data item stats - attachment 2


For further information see https://elog.ph.ed.ac.uk/AIDA/906 and https://elog.ph.ed.ac.uk/AIDA/907


To Do

- repair/replace Honeywell HSS-DPS dew point sensor
   USB-controlled ac mains relay interlock currently overrriden
   do NOT operate AIDA unattended

- aida04 asic #1 u/s - replace ASIC mezzanine

- electrically isolated test signal distribution box req'd

- aida10 asic #4
   v. high rates observed and large signal transients 
   cause unclear ... ASIC/adaptor PCB/cabling/Si wafer ?

- extended background alpha run to check all DSSSD bond wires
   pulser OFF
   slow comparator 0x64

- bPlas + 2x triple DSSSD + bPlas stack assembly and test

- all up, in beam test
Attachment 1: Screenshot_from_2023-10-12_15-13-30.png
Screenshot_from_2023-10-12_15-13-30.png
Attachment 2: Screenshot_from_2023-10-12_15-17-05.png
Screenshot_from_2023-10-12_15-17-05.png
  521   Sun Oct 1 11:47:07 2023 TDS4 cooling water
The photographs show the cooling water controls and temperature/pressure gauges outside S4 and the connections used by AIDA within S4.
Attachment 1: 1000007356.jpg
1000007356.jpg
Attachment 2: 1000007355.jpg
1000007355.jpg
  520   Thu Aug 31 15:24:56 2023 NHNew AIDA MBS PC
The AIDA MBS FDR will be x86l-119 from now on, not x86l-94

the MBS relay and startup scripts will be changed for this
  519   Mon Aug 28 12:47:56 2023 NHPower Failure 24.08.2023
There was a power failure in the morning of 24.08.2023 in the Rhein-Main area affecting GSI

The Aida workstation (aida-3) has been restarted, it is unknown if the Pis in S4 rebooted as well (there is a UPS)

29.8.23 TD Both RPi systems rebooted four days ago.
  518   Sun Jul 16 09:32:57 2023 TDFirefox browser proxy setting change
Firefox browser proxy setting changed to 'Auto-detect proxy settings for this network' ( Firefox -> Edit -> Settings -> Network Settings -> Settings )
  517   Wed Jan 18 13:40:33 2023 PJCS TDMACB settings with either Emulator or VITAR

When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.

This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.

When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.

Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings.

Attachment 1: macb_apr20.jed
Attachment 2: macb_apr20.vhd
----------------------------------------------------------------------------------
-- Company: 
-- Engineer: 
-- 
-- Create Date:    17:03:27 03/16/2011 
-- Design Name: 
-- Module Name:    macb_top - Behavioral 
-- Project Name: 
-- Target Devices: 
-- Tool versions: 
-- Description: 
--
-- Dependencies: 
--
-- Revision: 
-- Revision 0.01 - File Created
-- Additional Comments: 
--
----------------------------------------------------------------------------------
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.STD_LOGIC_ARITH.ALL;
use IEEE.STD_LOGIC_UNSIGNED.ALL;

---- Uncomment the following library declaration if instantiating
---- any Xilinx primitives in this code.
library UNISIM;
use UNISIM.VComponents.all;
-- NOTE all in/out notations are relative to this unit
entity macb_apr20 is
    Port ( 
			  port1_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port2_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port3_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port4_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  layer_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  layer_trigger : out std_logic ;
           sync_return : in  STD_LOGIC_VECTOR (3 downto 1);
           selector : in  STD_LOGIC_VECTOR (3 downto 0);
           sync_select : out  STD_LOGIC_vector(1 downto 0 );
           clock200_select : out  STD_LOGIC_vector( 1 downto 0 ) ;
			  butis_divide_reset : out std_logic ;
			  butis_divide_s : out std_logic_vector( 2 downto 0 ) ;
			  clock_5 : in std_logic ;
			  sync_5 : in std_logic ;
			  trigger : in std_logic_vector( 3 downto 0 ) ;
           MBS_in : in  STD_LOGIC_VECTOR (3 downto 0);
           MBS_out : out  STD_LOGIC_VECTOR (3 downto 0));
end macb_apr20;

architecture Behavioral of macb_apr20 is
signal port1_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port1_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port1_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port2_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port2_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port2_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port3_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port3_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port3_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port4_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port4_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port4_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal layer_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal layer_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal layer_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal seli : integer range 0 to 15 := 0  ;
-- well really
signal MBS_in_n : std_logic_vector( 3 downto 0 ) := "0000" ;
begin
MBS_in_n <= ( not MBS_in);
seli <= conv_integer(not selector) ;
-- MBS signal allocations to sp lines and HDMI pin. This maps to NIM connections
-- 0 :	MBS_clock10 	SP0	13
-- 1 :	MBS_reset		SP1	14
-- 2 :	MBS_reset_rq	SP2	15
-- 3 :	MBS_Trigger		SP3	16
layer_trigger <= trigger(0) or trigger(1) or trigger(2) or trigger(3) ;

-- divider controls set for pass-through
butis_divide_reset <= '1' ; -- for now don't reset ;

process ( seli , MBS_in_n, port1_spi, port2_spi, port3_spi, port4_spi, layer_spi, sync_return ,sync_5  )
-- note : & => concatenate
begin
	case seli is 
	when 0 => --- Master/ Root / MBS / Internal clock
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "00" ; -- select internal 200 MHz oscillator
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
		
	when 1 => --- Master/ Root / MBS / BuTiS clock and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 2 => --- Master/ Branch / MBS / Next layer clock next layer SYNC
		port1_spo <= layer_spi(3) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 3 => --- Slave / Branch / MBS / Next layer clock and sync
		port1_spo <= layer_spi(3) & '0'  & layer_spi(1) & layer_spi(0);
		port1_t <= "0100" ; -- drive clock, reset, trigger only 
		port2_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & '0' & '0' ; -- drive nothing
		layer_t <= "1111" ; -- just drive nothing down
		sync_select <= "10" ; -- select sync from next layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi ; -- map all the signals for monitoring ?
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
		
	when 4 => --- Master/ Root / MBS / BuTiS clock / Internal SYNC / External timestamp reset
		port1_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port1_t <= "0100" ; -- drive clock, reset, trigger only
		port2_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external 50 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & MBS_in_n(1) & sync_5 ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 5 => --- Master/ Root / MBS / External 50Mhz clock / Internal Sync
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external SMA input 
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass through.
			
	when 6 => --- Master/ Root / MBS / External 100Mhz clock / Internal Sync
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external SMA input 
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "100" ; -- s2 is 1 for external, 00 for /2.
		
	when 7 => --- Fast NIM input for each FEE / Next layer clock next layer SYNC
		port1_spo <= MBS_in_n(0) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(1) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(2) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 8 => --- Fast NIM input from Input 3 for each FEE / Next layer clock next layer SYNC
		port1_spo <= MBS_in_n(3) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
	
	when 9 => --- Master/ Root / Internal clock / sync_returns to NIM
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "00" ; -- select internal 200 MHz oscillator
		MBS_out <=  sync_return(3) & sync_return(2) & sync_return(1) & '0' ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 10 => --- Master/ Root / MBS / BuTiS clock /2 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= "0000" ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n ; -- for testing NIM I/O
		butis_divide_s <= "100" ; --  s2 = 1 and s1,s0 decode to 00=>/2 , 01=>/4, /8 , /16

	when 12 => --- Master/ Root / MBS / BuTiS clock /2 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "100" ; --  s2 = 1 and s1,s0 decode to 00=>/2 , 01=>/4, /8 , /16

	when 13 => --- Master/ Root / MBS / BuTiS clock /4 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "101" ; --  s2 = 1 and s1,s0 decode to  01=>/4
		
	when 14 => --- Master/ Root / MBS / BuTiS clock /8 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
... 161 more lines ...
Attachment 3: zybo.jpg
zybo.jpg
Attachment 4: MACB.jpg
MACB.jpg
  516   Fri Dec 16 14:02:12 2022 NHAIDA System off for christmas break
The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible
  515   Thu Oct 6 16:51:49 2022 NHOscilloscope analysis
Investigating AIDA noise with a TA041 differential probe and oscilloscope 

AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100

Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]

No significant noise or distortion present, fully within any AC specification. 
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
 No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)

Ion catcher not on (I think under repair)

-

FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10


Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]

Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]

Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?

Also see voltage changes with FEE power draw:

Power on  : 5.45 V (different scale to above)
SETUP ran : 5.51 V 
FADCs off : 5.86 V
ACQ Go:   : Unchanged; ASIC threshold 0xa: Unchanged

Also check situation on a fully loaded PSU (8 fees connected and powered on)

Power on:   5.29 V (fig 8)
SETUP ran:  5.36 V (fig 9)
FADCs off:  5.64 V (fig 10)

All X: 500 ns/div, Y: 100 mV/div

Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)

Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)

No sign of strong 100 kHz noise as seen in ADC traces beforehand

Todo:

- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds 
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side

-
11.10.22 Updates

Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour

Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region

Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively

12.10.22 Updates

Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V

Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V

Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only

Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...


Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with  no FEEs connected to PSU
 See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
 Looks to be common mode noise (on both 5V and Return of PSU)

Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present
Attachment 1: SCRN0086.PNG
SCRN0086.PNG
Attachment 2: SCRN0090.jpg
SCRN0090.jpg
Attachment 3: SCRN0113.jpg
SCRN0113.jpg
Attachment 4: SCRN0107.jpg
SCRN0107.jpg
Attachment 5: SCRN0108.jpg
SCRN0108.jpg
Attachment 6: Image_Pasted_at_2022-10-6_15-51.jpg
Image_Pasted_at_2022-10-6_15-51.jpg
Attachment 7: SCRN0098.jpg
SCRN0098.jpg
Attachment 8: SCRN0109.jpg
SCRN0109.jpg
Attachment 9: SCRN0110.jpg
SCRN0110.jpg
Attachment 10: SCRN0111.jpg
SCRN0111.jpg
Attachment 11: SCRN0115.PNG
SCRN0115.PNG
Attachment 12: SCRN0121.PNG
SCRN0121.PNG
Attachment 13: SCRN0124.PNG
SCRN0124.PNG
Attachment 14: SCRN0125.PNG
SCRN0125.PNG
Attachment 15: SCRN0126.PNG
SCRN0126.PNG
Attachment 16: SCRN0127.PNG
SCRN0127.PNG
Attachment 17: SCRN0132.PNG
SCRN0132.PNG
Attachment 18: SCRN0129.PNG
SCRN0129.PNG
Attachment 19: SCRN0139.PNG
SCRN0139.PNG
Attachment 20: SCRN0136.PNG
SCRN0136.PNG
Attachment 21: SCRN0141.PNG
SCRN0141.PNG
Attachment 22: SCRN0143.PNG
SCRN0143.PNG
Attachment 23: SCRN0149.PNG
SCRN0149.PNG
Attachment 24: SCRN0153.PNG
SCRN0153.PNG
  514   Wed Sep 14 19:07:07 2022 PJCSINFO : Three Merger Statistics explained

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

 

  513   Thu Sep 8 12:37:18 2022 NHProxy Port Changed
The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use
  512   Thu Sep 8 12:31:25 2022 NHRetrying AIDA DataAcq v10
Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 
Attachment 1: AnyDeskMSI_2022-09-08_13-35-50.png
AnyDeskMSI_2022-09-08_13-35-50.png
Attachment 2: AnyDeskMSI_2022-09-08_13-35-58.png
AnyDeskMSI_2022-09-08_13-35-58.png
Attachment 3: AnyDeskMSI_2022-09-08_13-36-02.png
AnyDeskMSI_2022-09-08_13-36-02.png
  511   Tue Aug 30 13:48:32 2022 NHAIDA Single Switch Configuration
The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system
  510   Tue Jul 5 08:53:20 2022 TDTo Do
In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   
  509   Wed Jun 29 10:48:26 2022 NH, OHAIDA Dismounted
All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 
  508   Tue Jun 28 10:11:35 2022 OH, NHMIDAS Data Aq V10
11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?
Attachment 1: 220628_1117_Rollover0xE.png
220628_1117_Rollover0xE.png
Attachment 2: 220628_1119_Merger_Stats_0x7.png
220628_1119_Merger_Stats_0x7.png
Attachment 3: NewMerger_Dump.txt
MERGE Data Link (17671): block sequence error: ID 2, expected 462; received 0
MERGE Data Link (17669): block sequence error: ID 0, expected 461; received 0
MERGE Data Link (17672): block sequence error: ID 3, expected 462; received 0
MERGE Data Link (17670): block sequence error: ID 1, expected 462; received 0
MERGE Data Link (17674): block sequence error: ID 5, expected 463; received 0
MERGE Data Link (17676): block sequence error: ID 7, expected 462; received 0
MERGE Data Link (17675): block sequence error: ID 6, expected 461; received 0
MERGE Data Link (17673): block sequence error: ID 4, expected 462; received 0
MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

  507   Tue Jun 28 09:37:19 2022 NHTues 28 June 08:00-
Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after
  506   Mon Jun 27 23:11:47 2022 TDTuesday 28 June 00:00-08:00
00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30
Attachment 1: R5_896
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261821326 (  595374.9 Hz)
        Other data format:      98676 (     224.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        406 (       0.9 Hz)
                               RESUME:        406 (       0.9 Hz)
                              SYNC100:      33638 (      76.5 Hz)
                              WR48-63:      33638 (      76.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      30588 (      69.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      439.759 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.599                0.000
  2                0.000                0.000
  3                0.043                0.000
  4                0.000              153.964
  5                0.000                0.000
  6                0.053                0.000
  7                0.015                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19090298      35782          0          0          0          0       2597       2597          0      30588          0          0
  1   70343282      19202          0          0        400        400       9201       9201          0          0          0          0
  2   10361360       2782          0          0          0          0       1391       1391          0          0          0          0
  3   39578798      10102          0          0          3          3       5048       5048          0          0          0          0
  4    3065688        742          0          0          0          0        371        371          0          0          0          0
  5   15482560       3946          0          0          0          0       1973       1973          0          0          0          0
  6   51969876      12870          0          0          2          2       6433       6433          0          0          0          0
  7   51929464      13250          0          0          1          1       6624       6624          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.371s (  904.697 blocks/s,  56.544 Mb/s)
Attachment 2: Screenshot_from_2022-06-28_00-06-58.png
Screenshot_from_2022-06-28_00-06-58.png
Attachment 3: Screenshot_from_2022-06-28_00-06-35.png
Screenshot_from_2022-06-28_00-06-35.png
Attachment 4: Screenshot_from_2022-06-28_00-06-00.png
Screenshot_from_2022-06-28_00-06-00.png
Attachment 5: Screenshot_from_2022-06-28_00-05-30.png
Screenshot_from_2022-06-28_00-05-30.png
Attachment 6: Screenshot_from_2022-06-28_00-04-42.png
Screenshot_from_2022-06-28_00-04-42.png
Attachment 7: Screenshot_from_2022-06-28_00-03-56.png
Screenshot_from_2022-06-28_00-03-56.png
Attachment 8: R5_914
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261460369 (  646962.6 Hz)
        Other data format:     459631 (    1137.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        616 (       1.5 Hz)
                               RESUME:        616 (       1.5 Hz)
                              SYNC100:      33505 (      82.9 Hz)
                              WR48-63:      33505 (      82.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     391389 (     968.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     131720 (     325.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      404.135 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.615                0.000
  1                2.119                0.000
  2                0.420                0.000
  3                0.323                0.000
  4                0.124              111.840
  5                0.000                0.000
  6                0.696                0.000
  7                0.001                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19284409      33077          0          0          6          6       2512       2512          0      28041          0      31815
  1   69379431      18886          0          0        391        391       9052       9052          0          0          0      26209
  2   20984122      96945          0          0          6          6       2775       2775          0      91383          0      19775
  3   56761004     105837          0          0        202        202       7127       7127          0      91179          0       9889
  4    3261224        838          0          0          2          2        417        417          0          0          0       9818
  5   13707273       3434          0          0          0          0       1717       1717          0          0          0      11918
  6   42321326     100064          0          0          8          8       5351       5351          0      89346          0      17237
  7   35761580     100550          0          0          1          1       4554       4554          0      91440          0       5059
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.648s (  897.667 blocks/s,  56.104 Mb/s)
Attachment 9: Screenshot_from_2022-06-28_02-14-50.png
Screenshot_from_2022-06-28_02-14-50.png
Attachment 10: Screenshot_from_2022-06-28_02-13-46.png
Screenshot_from_2022-06-28_02-13-46.png
Attachment 11: Screenshot_from_2022-06-28_02-12-22.png
Screenshot_from_2022-06-28_02-12-22.png
Attachment 12: Screenshot_from_2022-06-28_02-11-10.png
Screenshot_from_2022-06-28_02-11-10.png
Attachment 13: Screenshot_from_2022-06-28_02-10-07.png
Screenshot_from_2022-06-28_02-10-07.png
Attachment 14: Screenshot_from_2022-06-28_02-08-49.png
Screenshot_from_2022-06-28_02-08-49.png
Attachment 15: Screenshot_from_2022-06-28_02-05-26.png
Screenshot_from_2022-06-28_02-05-26.png
Attachment 16: Screenshot_from_2022-06-28_02-04-36.png
Screenshot_from_2022-06-28_02-04-36.png
Attachment 17: Screenshot_from_2022-06-28_02-04-11.png
Screenshot_from_2022-06-28_02-04-11.png
Attachment 18: Screenshot_from_2022-06-28_02-03-39.png
Screenshot_from_2022-06-28_02-03-39.png
Attachment 19: R5_926
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261820610 (  589706.0 Hz)
        Other data format:      99390 (     223.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        624 (       1.4 Hz)
                               RESUME:        624 (       1.4 Hz)
                              SYNC100:      33654 (      75.8 Hz)
                              WR48-63:      33654 (      75.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      30834 (      69.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      443.985 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.081                0.000
  2                0.000                0.000
  3                0.281                0.000
  4                0.000              152.429
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15622398      34978          0          0          0          0       2072       2072          0      30834          0          0
  1   68236330      18574          0          0        381        381       8906       8906          0          0          0          0
  2   18744892       4984          0          0          0          0       2492       2492          0          0          0          0
  3   59875268      15514          0          0        243        243       7514       7514          0          0          0          0
  4    2759883        664          0          0          0          0        332        332          0          0          0          0
  5   13541412       3478          0          0          0          0       1739       1739          0          0          0          0
  6   45470554      11726          0          0          0          0       5863       5863          0          0          0          0
  7   37569873       9472          0          0          0          0       4736       4736          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.045s (  913.114 blocks/s,  57.070 Mb/s)
Attachment 20: Screenshot_from_2022-06-28_03-44-20.png
Screenshot_from_2022-06-28_03-44-20.png
Attachment 21: Screenshot_from_2022-06-28_03-44-00.png
Screenshot_from_2022-06-28_03-44-00.png
Attachment 22: Screenshot_from_2022-06-28_03-43-05.png
Screenshot_from_2022-06-28_03-43-05.png
Attachment 23: Screenshot_from_2022-06-28_03-41-50.png
Screenshot_from_2022-06-28_03-41-50.png
Attachment 24: Screenshot_from_2022-06-28_03-41-21.png
Screenshot_from_2022-06-28_03-41-21.png
Attachment 25: R5_954
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261819650 (  564425.6 Hz)
        Other data format:     100350 (     216.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        346 (       0.7 Hz)
                               RESUME:        346 (       0.7 Hz)
                              SYNC100:      33728 (      72.7 Hz)
                              WR48-63:      33728 (      72.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      32202 (      69.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      463.869 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.860                0.000
  2                0.000               16.297
  3                0.000                0.000
  4                0.000              130.996
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19510795      37524          0          0          0          0       2661       2661          0      32202          0          0
  1   71776267      19370          0          0        343        343       9342       9342          0          0          0          0
  2    7596519       1752          0          0          0          0        876        876          0          0          0          0
  3   39174917      10150          0          0          0          0       5075       5075          0          0          0          0
  4    3312486        838          0          0          0          0        419        419          0          0          0          0
  5   15419570       4004          0          0          0          0       2002       2002          0          0          0          0
  6   56857618      14472          0          0          3          3       7233       7233          0          0          0          0
  7   48171478      12240          0          0          0          0       6120       6120          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.318s (  906.044 blocks/s,  56.628 Mb/s)
Attachment 26: Screenshot_from_2022-06-28_07-03-01.png
Screenshot_from_2022-06-28_07-03-01.png
Attachment 27: Screenshot_from_2022-06-28_07-02-34.png
Screenshot_from_2022-06-28_07-02-34.png
Attachment 28: Screenshot_from_2022-06-28_07-01-31.png
Screenshot_from_2022-06-28_07-01-31.png
Attachment 29: Screenshot_from_2022-06-28_07-01-03.png
Screenshot_from_2022-06-28_07-01-03.png
Attachment 30: Screenshot_from_2022-06-28_07-00-31.png
Screenshot_from_2022-06-28_07-00-31.png
  505   Mon Jun 27 17:03:09 2022 MAMonday 27th June 16:00-00:00

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

Attachment 1: Statistics2022-06-27_18-02-25.png
Statistics2022-06-27_18-02-25.png
Attachment 2: Temperature2022-06-27_18-01-14.png
Temperature2022-06-27_18-01-14.png
Attachment 3: Current2022-06-27_17-59-55.png
Current2022-06-27_17-59-55.png
Attachment 4: Statics2022-06-27_22-12-07.png
Statics2022-06-27_22-12-07.png
Attachment 5: Temperature2022-06-27_22-11-21.png
Temperature2022-06-27_22-11-21.png
Attachment 6: Current2022-06-27_22-10-12.png
Current2022-06-27_22-10-12.png
  504   Mon Jun 27 06:45:22 2022 OHMonday 27th June 08:00-16:00
07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?
Attachment 1: 220626_0830_Stats.png
220626_0830_Stats.png
Attachment 2: 220626_0830_Temp.png
220626_0830_Temp.png
Attachment 3: 220626_0831_Bias.png
220626_0831_Bias.png
Attachment 4: R5_770_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261823568 (  639881.5 Hz)
        Other data format:      96432 (     235.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        451 (       1.1 Hz)
                               RESUME:        451 (       1.1 Hz)
                              SYNC100:      33524 (      81.9 Hz)
                              WR48-63:      33524 (      81.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      28482 (      69.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      409.175 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.033                0.000
  1                1.307                0.000
  2                0.000                0.000
  3                0.206                0.000
  4                0.000              157.882
  5                0.000                0.000
  6                0.000                0.000
  7                0.139                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20319054      33950          0          0          1          1       2733       2733          0      28482          0          0
  1   68104715      18442          0          0        318        318       8903       8903          0          0          0          0
  2   24150534       6320          0          0          0          0       3160       3160          0          0          0          0
  3   55928626      14410          0          0        131        131       7074       7074          0          0          0          0
  4    2503858        618          0          0          0          0        309        309          0          0          0          0
  5   12980974       3340          0          0          0          0       1670       1670          0          0          0          0
  6   41271038      10344          0          0          0          0       5172       5172          0          0          0          0
  7   36564769       9008          0          0          1          1       4503       4503          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.158s (  777.488 blocks/s,  48.593 Mb/s)
  503   Sun Jun 26 23:04:30 2022 Marcnew shift - Monday 27 June 0:00 to 8:00

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

Attachment 1: Stats-Screenshot_from_2022-06-27_00-30-58.png
Stats-Screenshot_from_2022-06-27_00-30-58.png
Attachment 2: Temp-Screenshot_from_2022-06-27_00-31-44.png
Temp-Screenshot_from_2022-06-27_00-31-44.png
Attachment 3: HV-LC-Screenshot_from_2022-06-27_00-30-18.png
HV-LC-Screenshot_from_2022-06-27_00-30-18.png
Attachment 4: Stats-Screenshot_from_2022-06-27_02-22-34.png
Stats-Screenshot_from_2022-06-27_02-22-34.png
Attachment 5: Temp-Screenshot_from_2022-06-27_02-19-42.png
Temp-Screenshot_from_2022-06-27_02-19-42.png
Attachment 6: HV-LC-Screenshot_from_2022-06-27_02-19-05.png
HV-LC-Screenshot_from_2022-06-27_02-19-05.png
Attachment 7: Stats-Screenshot_from_2022-06-27_04-17-22.png
Stats-Screenshot_from_2022-06-27_04-17-22.png
Attachment 8: Temp-Screenshot_from_2022-06-27_04-16-40.png
Temp-Screenshot_from_2022-06-27_04-16-40.png
Attachment 9: HV-LC-Screenshot_from_2022-06-27_04-14-57.png
HV-LC-Screenshot_from_2022-06-27_04-14-57.png
Attachment 10: Stats-Screenshot_from_2022-06-27_07-15-17.png
Stats-Screenshot_from_2022-06-27_07-15-17.png
Attachment 11: Temp-Screenshot_from_2022-06-27_07-14-02.png
Temp-Screenshot_from_2022-06-27_07-14-02.png
Attachment 12: HV-LC-Screenshot_from_2022-06-27_07-13-09.png
HV-LC-Screenshot_from_2022-06-27_07-13-09.png
  502   Sun Jun 26 08:51:20 2022 OH, NHSunday 26 June 08:00-24:00
09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27
Attachment 1: 220626_0949_stats.png
220626_0949_stats.png
Attachment 2: 220626_0950_Temps.png
220626_0950_Temps.png
Attachment 3: 220626_0951_Bias.png
220626_0951_Bias.png
Attachment 4: R5_528_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260639000 (  776334.1 Hz)
        Other data format:    1281002 (    3815.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1133 (       3.4 Hz)
                               RESUME:       1133 (       3.4 Hz)
                              SYNC100:      33251 (      99.0 Hz)
                              WR48-63:      33251 (      99.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1212234 (    3610.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     393457 (    1171.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      335.730 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                8.363                0.000
  1               44.893                0.000
  2                0.179                0.000
  3                3.779                0.000
  4                0.000               44.522
  5                0.034                0.000
  6                0.166                0.000
  7                0.017                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31317521      31305          0          0        207        207       4035       4035          0      22821          0      98396
  1   68454688      19188          0          0        693        693       8901       8901          0          0          0      59818
  2   19369571     305849          0          0         35         35       2468       2468          0     300843          0      63063
  3   45019317     306201          0          0        130        130       5636       5636          0     294669          0      32623
  4    4769649       1252          0          0          0          0        626        626          0          0          0      31000
  5   17853070       4600          0          0         15         15       2285       2285          0          0          0      37328
  6   42448852     303192          0          0         44         44       5356       5356          0     292392          0      54344
  7   31406332     309415          0          0          9          9       3944       3944          0     301509          0      16885
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.891s (  763.894 blocks/s,  47.743 Mb/s)
Attachment 5: 220626_1155_stats.png
220626_1155_stats.png
Attachment 6: 220626_1155_Temp.png
220626_1155_Temp.png
Attachment 7: 220626_1156_Bias.png
220626_1156_Bias.png
Attachment 8: R5_548_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260758427 (  740797.4 Hz)
        Other data format:    1161573 (    3300.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1084 (       3.1 Hz)
                               RESUME:       1084 (       3.1 Hz)
                              SYNC100:      33311 (      94.6 Hz)
                              WR48-63:      33311 (      94.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1092783 (    3104.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     356241 (    1012.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      351.997 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.340                0.000
  1               40.660                0.000
  2                0.175                0.000
  3                2.385                0.000
  4                0.000               67.319
  5                0.032                0.000
  6                0.142                0.000
  7                0.019                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29258900      32253          0          0        166        166       3875       3875          0      24171          0      88948
  1   70599060      19658          0          0        716        716       9113       9113          0          0          0      56350
  2   19313501     275533          0          0         28         28       2458       2458          0     270561          0      56248
  3   44887469     278330          0          0        112        112       5682       5682          0     266742          0      29387
  4    4430708       1118          0          0          0          0        559        559          0          0          0      27825
  5   17540711       4456          0          0         11         11       2217       2217          0          0          0      33663
  6   43048879     271297          0          0         40         40       5377       5377          0     260463          0      48745
  7   31679199     278928          0          0         11         11       4030       4030          0     270846          0      15075
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   40.309s (  793.875 blocks/s,  49.617 Mb/s)
Attachment 9: 220626_1435_stats.png
220626_1435_stats.png
Attachment 10: 220626_1435_Temp.png
220626_1435_Temp.png
Attachment 11: 220626_1436_Bias.png
220626_1436_Bias.png
Attachment 12: 220626_1620_Stats.png
220626_1620_Stats.png
Attachment 13: 220627_1620_Temp.png
220627_1620_Temp.png
Attachment 14: 220626_1621_Bias.png
220626_1621_Bias.png
Attachment 15: R5_591_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260421862 (  840783.5 Hz)
        Other data format:    1498138 (    4836.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1449 (       4.7 Hz)
                               RESUME:       1448 (       4.7 Hz)
                              SYNC100:      33154 (     107.0 Hz)
                              WR48-63:      33154 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1428933 (    4613.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     452287 (    1460.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.737 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               13.828                0.000
  1               52.434                0.000
  2                0.366                0.000
  3                9.346                0.000
  4                0.001               63.918
  5                0.097                0.000
  6                0.290                0.000
  7                0.023                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31248979      29393          0          0        283        282       4085       4085          0      20658          0     112900
  1   66285610      18734          0          0        794        794       8573       8573          0          0          0      62107
  2   22411069     363765          0          0         53         53       2890       2890          0     357879          0      73563
  3   44202913     357046          0          0        221        221       5556       5556          0     345492          0      37941
  4    5178577       1284          0          0          1          1        641        641          0          0          0      37680
  5   19493707       4858          0          0         26         26       2403       2403          0          0          0      44391
  6   39593301     355717          0          0         57         57       5081       5081          0     345441          0      63694
  7   32007706     367341          0          0         14         14       3925       3925          0     359463          0      20011
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.789s (  765.751 blocks/s,  47.859 Mb/s)
Attachment 16: tmp_aida_stats.png
tmp_aida_stats.png
Attachment 17: tmp_aida_temps.png
tmp_aida_temps.png
Attachment 18: tmp_aida_hv.png
tmp_aida_hv.png
Attachment 19: R5_620
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261173934 (  672570.7 Hz)
        Other data format:     746069 (    1921.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        773 (       2.0 Hz)
                               RESUME:        773 (       2.0 Hz)
                              SYNC100:      33446 (      86.1 Hz)
                              WR48-63:      33446 (      86.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     677631 (    1745.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     225736 (     581.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      388.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.039                0.000
  1               21.090                0.000
  2                0.051                0.000
  3                0.148                0.000
  4                0.000               87.597
  5                0.001                0.000
  6                0.083                0.000
  7                0.003                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   25216948      33824          0          0         63         63       3334       3334          0      27030          0      55557
  1   75946108      20872          0          0        626        626       9810       9810          0          0          0      40146
  2   18060200     168976          0          0         23         23       2304       2304          0     164322          0      34386
  3   41452092     174351          0          0         26         26       5287       5287          0     163725          0      18377
  4    3680704        948          0          0          0          0        474        474          0          0          0      17387
  5   16518678       4150          0          0          1          1       2074       2074          0          0          0      20823
  6   44456577     169523          0          0         30         30       5686       5686          0     158091          0      29842
  7   35842627     173425          0          0          4          4       4477       4477          0     164463          0       9218
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.547s (  770.214 blocks/s,  48.138 Mb/s)
Attachment 20: 220624_2021_Stats.png
220624_2021_Stats.png
Attachment 21: 220626_2021_Temp.png
220626_2021_Temp.png
Attachment 22: 220626_2022_Bias.png
220626_2022_Bias.png
Attachment 23: R5_637_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260438015 (  849636.0 Hz)
        Other data format:    1481985 (    4834.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1460 (       4.8 Hz)
                               RESUME:       1460 (       4.8 Hz)
                              SYNC100:      33142 (     108.1 Hz)
                              WR48-63:      33142 (     108.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1412781 (    4609.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     447675 (    1460.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      306.529 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               15.951                0.000
  1               53.516                0.000
  2                0.642                0.000
  3                7.248                0.000
  4                0.000               53.241
  5                0.100                0.000
  6                0.344                0.000
  7                0.040                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31824689      29138          0          0        291        291       4129       4129          0      20298          0     110511
  1   66116092      18806          0          0        788        788       8615       8615          0          0          0      62706
  2   22609638     359670          0          0         68         68       2899       2899          0     353736          0      72536
  3   44231727     352857          0          0        199        199       5696       5696          0     341067          0      37873
  4    5159378       1270          0          0          0          0        635        635          0          0          0      37425
  5   19491876       4836          0          0         27         27       2391       2391          0          0          0      43832
  6   39561529     352144          0          0         68         68       4899       4899          0     342210          0      62927
  7   31443086     363264          0          0         19         19       3878       3878          0     355470          0      19865
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.891s (  746.084 blocks/s,  46.630 Mb/s)
Attachment 24: 220626_2211_Stats.png
220626_2211_Stats.png
Attachment 25: 220626_2211_Temps.png
220626_2211_Temps.png
Attachment 26: 220626_2212_Bias.png
220626_2212_Bias.png
Attachment 27: R5_658_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260445182 (  832450.2 Hz)
        Other data format:    1474818 (    4713.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1484 (       4.7 Hz)
                               RESUME:       1484 (       4.7 Hz)
                              SYNC100:      33166 (     106.0 Hz)
                              WR48-63:      33166 (     106.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1405518 (    4492.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     448659 (    1434.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               15.080                0.000
  1               57.798                0.000
  2                0.343                0.000
  3                7.278                0.000
  4                0.000               44.445
  5                0.093                0.000
  6                0.265                0.000
  7                0.017                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   32528426      29869          0          0        310        310       4234       4234          0      20781          0     110841
  1   66604779      18608          0          0        825        825       8479       8479          0          0          0      61991
  2   23919816     357449          0          0         63         63       3136       3136          0     351051          0      72944
  3   47288432     353667          0          0        196        196       5939       5939          0     341397          0      38095
  4    5032393       1218          0          0          0          0        609        609          0          0          0      37603
  5   19089927       4824          0          0         28         28       2384       2384          0          0          0      44213
  6   39392581     350027          0          0         52         52       5076       5076          0     339771          0      63206
  7   26588828     359156          0          0         10         10       3309       3309          0     352518          0      19766
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.102s (  760.067 blocks/s,  47.504 Mb/s)
  501   Sat Jun 25 22:48:57 2022 TDSunday 26 June 00:00-08:00
23.43 Check ASIC control
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2

DSSSD bias & leakage currents OK - attachment 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6

analysis of file S505/R5_415 - attachment 7
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 1%



02:36

analysis of file S505/R5_449 - attachment 8
 zero timewarps
 max deadtime aida02 c. 15%, all other FEE64s < 1%

per FEE64 rate & stat spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 99 ch FWHM

per FEE64 1.8.H spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


05:38

analysis of file S505/R5_483 - attachment 19
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24



07:27

analysis of file S505/R5_502 - attachment 25
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30
Attachment 1: Screenshot_from_2022-06-25_23-48-13.png
Screenshot_from_2022-06-25_23-48-13.png
Attachment 2: Screenshot_from_2022-06-25_23-47-49.png
Screenshot_from_2022-06-25_23-47-49.png
Attachment 3: Screenshot_from_2022-06-25_23-46-30.png
Screenshot_from_2022-06-25_23-46-30.png
Attachment 4: Screenshot_from_2022-06-25_23-45-38.png
Screenshot_from_2022-06-25_23-45-38.png
Attachment 5: Screenshot_from_2022-06-25_23-43-15.png
Screenshot_from_2022-06-25_23-43-15.png
Attachment 6: Screenshot_from_2022-06-25_23-42-41.png
Screenshot_from_2022-06-25_23-42-41.png
Attachment 7: R5_415
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260603334 (  839394.5 Hz)
        Other data format:    1316666 (    4240.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1333 (       4.3 Hz)
                               RESUME:       1333 (       4.3 Hz)
                              SYNC100:      33156 (     106.8 Hz)
                              WR48-63:      33156 (     106.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1247688 (    4018.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     404733 (    1303.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      310.466 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.804                0.000
  1               49.047                0.000
  2                0.579                0.000
  3                4.786                0.000
  4                0.000               41.193
  5                0.052                0.000
  6                0.398                0.000
  7                0.178                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29744429      29368          0          0        191        191       3879       3879          0      21228          0     101774
  1   62881123      17488          0          0        744        744       8000       8000          0          0          0      60446
  2   27391657     317618          0          0         76         76       3507       3507          0     310452          0      64539
  3   52390232     315971          0          0        228        228       6571       6571          0     302373          0      33153
  4    4668386       1144          0          0          0          0        572        572          0          0          0      32450
  5   16955242       4388          0          0         19         19       2175       2175          0          0          0      39119
  6   36767424     310864          0          0         60         60       4661       4661          0     301422          0      55985
  7   29804841     319825          0          0         15         15       3791       3791          0     312213          0      17267
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.203s (  758.238 blocks/s,  47.390 Mb/s)
Attachment 8: R5_449
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9631
          ADC data format:   78467361 (  817544.9 Hz)
        Other data format:     362374 (    3775.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        372 (       3.9 Hz)
                               RESUME:        372 (       3.9 Hz)
                              SYNC100:       9989 (     104.1 Hz)
                              WR48-63:       9989 (     104.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     341652 (    3559.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     112408 (    1171.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       95.979 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.159                0.000
  1               13.633                0.000
  2                0.066                0.000
  3                0.790                0.000
  4                0.000               20.217
  5                0.003                0.000
  6                0.066                0.000
  7                0.003                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8759445       9043          0          0         45         45       1175       1175          0       6603          0      28377
  1   19225571       5374          0          0        215        215       2472       2472          0          0          0      17713
  2    7917505      86814          0          0         15         15       1017       1017          0      84750          0      17600
  3   15877732      87394          0          0         75         75       1976       1976          0      83292          0       9310
  4    1340586        302          0          0          0          0        151        151          0          0          0       8860
  5    4870076       1158          0          0          2          2        577        577          0          0          0      10559
  6   11177308      84854          0          0         17         17       1400       1400          0      82020          0      15171
  7    9299138      87435          0          0          3          3       1221       1221          0      84987          0       4818
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.232s (  941.224 blocks/s,  58.826 Mb/s)
Attachment 9: Screenshot_from_2022-06-26_02-36-29.png
Screenshot_from_2022-06-26_02-36-29.png
Attachment 10: Screenshot_from_2022-06-26_02-34-58.png
Screenshot_from_2022-06-26_02-34-58.png
Attachment 11: Screenshot_from_2022-06-26_02-33-46.png
Screenshot_from_2022-06-26_02-33-46.png
Attachment 12: Screenshot_from_2022-06-26_02-32-48.png
Screenshot_from_2022-06-26_02-32-48.png
Attachment 13: Screenshot_from_2022-06-26_02-31-49.png
Screenshot_from_2022-06-26_02-31-49.png
Attachment 14: Screenshot_from_2022-06-26_02-30-49.png
Screenshot_from_2022-06-26_02-30-49.png
Attachment 15: Screenshot_from_2022-06-26_02-30-25.png
Screenshot_from_2022-06-26_02-30-25.png
Attachment 16: Screenshot_from_2022-06-26_02-29-31.png
Screenshot_from_2022-06-26_02-29-31.png
Attachment 17: Screenshot_from_2022-06-26_02-28-59.png
Screenshot_from_2022-06-26_02-28-59.png
Attachment 18: Screenshot_from_2022-06-26_02-28-31.png
Screenshot_from_2022-06-26_02-28-31.png
Attachment 19: R5_483
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260645305 (  834521.2 Hz)
        Other data format:    1274695 (    4081.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1211 (       3.9 Hz)
                               RESUME:       1211 (       3.9 Hz)
                              SYNC100:      33162 (     106.2 Hz)
                              WR48-63:      33162 (     106.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1205949 (    3861.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     393822 (    1260.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.329 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.184                0.000
  1               48.274                0.000
  2                0.431                0.000
  3                3.375                0.000
  4                0.000               25.625
  5                0.027                0.000
  6                0.216                0.000
  7                0.008                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30200144      29499          0          0        166        166       3899       3899          0      21369          0      99521
  1   63245692      17566          0          0        710        710       8073       8073          0          0          0      60306
  2   25915590     306468          0          0         52         52       3383       3383          0     299598          0      62507
  3   52691198     306273          0          0        212        212       6559       6559          0     292731          0      32107
  4    4611863       1212          0          0          0          0        606        606          0          0          0      31071
  5   16567070       4140          0          0          9          9       2061       2061          0          0          0      37441
  6   37080833     301098          0          0         53         53       4819       4819          0     291354          0      54190
  7   30332915     308439          0          0          9          9       3762       3762          0     300897          0      16679
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.023s (  913.674 blocks/s,  57.105 Mb/s)
Attachment 20: Screenshot_from_2022-06-26_05-40-48.png
Screenshot_from_2022-06-26_05-40-48.png
Attachment 21: Screenshot_from_2022-06-26_05-40-20.png
Screenshot_from_2022-06-26_05-40-20.png
Attachment 22: Screenshot_from_2022-06-26_05-39-34.png
Screenshot_from_2022-06-26_05-39-34.png
Attachment 23: Screenshot_from_2022-06-26_05-38-30.png
Screenshot_from_2022-06-26_05-38-30.png
Attachment 24: Screenshot_from_2022-06-26_05-37-55.png
Screenshot_from_2022-06-26_05-37-55.png
Attachment 25: R5_502
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260592349 (  788577.6 Hz)
        Other data format:    1327651 (    4017.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1258 (       3.8 Hz)
                               RESUME:       1257 (       3.8 Hz)
                              SYNC100:      33231 (     100.6 Hz)
                              WR48-63:      33231 (     100.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1258674 (    3808.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     409569 (    1239.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      330.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.444                0.000
  1               49.479                0.000
  2                0.188                0.000
  3                4.232                0.000
  4                0.000               57.929
  5                0.053                0.000
  6                0.278                0.000
  7                0.015                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31755884      31086          0          0        242        242       4168       4168          0      22266          0     103011
  1   67198908      18739          0          0        724        723       8646       8646          0          0          0      63328
  2   19438169     317456          0          0         45         45       2440       2440          0     312486          0      65069
  3   45255369     318470          0          0        158        158       5750       5750          0     306654          0      33811
  4    4887217       1230          0          0          0          0        615        615          0          0          0      32106
  5   18040993       4688          0          0         19         19       2325       2325          0          0          0      38770
  6   42367504     314874          0          0         58         58       5348       5348          0     304062          0      56174
  7   31648305     321108          0          0         12         12       3939       3939          0     313206          0      17300
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.316s (  960.488 blocks/s,  60.030 Mb/s)
Attachment 26: Screenshot_from_2022-06-26_07-29-14.png
Screenshot_from_2022-06-26_07-29-14.png
Attachment 27: Screenshot_from_2022-06-26_07-28-54.png
Screenshot_from_2022-06-26_07-28-54.png
Attachment 28: Screenshot_from_2022-06-26_07-28-22.png
Screenshot_from_2022-06-26_07-28-22.png
Attachment 29: Screenshot_from_2022-06-26_07-27-50.png
Screenshot_from_2022-06-26_07-27-50.png
Attachment 30: Screenshot_from_2022-06-26_07-27-21.png
Screenshot_from_2022-06-26_07-27-21.png
  500   Sat Jun 25 17:06:42 2022 BA, AASaturday 25 June 2022 16:00-00:00

Took over the night shift from Magda

 

18:00 : attachments 1-3

20:00 : attachments 4-6

22:00 : attachments 7-9

00:00 : attachments 10-12

 

Attachment 1: Stat_17.png
Stat_17.png
Attachment 2: leak_17.png
leak_17.png
Attachment 3: Temp17.png
Temp17.png
Attachment 4: Temp_20.png
Temp_20.png
Attachment 5: Stat_20.png
Stat_20.png
Attachment 6: lek_20.png
lek_20.png
Attachment 7: stat_22.png
stat_22.png
Attachment 8: temp_22.png
temp_22.png
Attachment 9: lak_22.png
lak_22.png
  499   Sat Jun 25 06:55:07 2022 MSSaturday 25 June 2022 8:00-16:00

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15

 

 

Attachment 1: Screenshot_2022-06-25_at_08-01-16_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_08-01-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-06-25_at_08-01-37_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_08-01-37_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-06-25_08-02-58.png
Screenshot_from_2022-06-25_08-02-58.png
Attachment 4: Screenshot_2022-06-25_at_10-00-41_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_10-00-41_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-06-25_at_10-01-03_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_10-01-03_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-06-25_10-01-59.png
Screenshot_from_2022-06-25_10-01-59.png
Attachment 7: Screenshot_2022-06-25_at_12-00-54_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_12-00-54_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2022-06-25_at_12-01-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_12-01-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_from_2022-06-25_12-02-08.png
Screenshot_from_2022-06-25_12-02-08.png
Attachment 10: Screenshot_2022-06-25_at_14-07-36_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_14-07-36_Statistics_aidas-gsi.png
Attachment 11: Screenshot_2022-06-25_at_14-07-59_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_14-07-59_Temperature_and_status_scan_aidas-gsi.png
Attachment 12: Screenshot_from_2022-06-25_14-08-50.png
Screenshot_from_2022-06-25_14-08-50.png
Attachment 13: Screenshot_2022-06-25_at_15-56-41_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_15-56-41_Statistics_aidas-gsi.png
Attachment 14: Screenshot_2022-06-25_at_15-56-57_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_15-56-57_Temperature_and_status_scan_aidas-gsi.png
Attachment 15: Screenshot_from_2022-06-25_15-58-15.png
Screenshot_from_2022-06-25_15-58-15.png
  498   Fri Jun 24 22:36:45 2022 TDSaturday 25 June 00:00-08:00
23:36 Check ASIC contorol
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_154 - attachment 1
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

stats - attachments 2-6

per FEE64 1.8.H spectra - attachments 7 & 8

per p+n FEE64 1.8.L spectra - attachment 9
 aida01 pulser peak width 101 ch FWHM

per FEE64 stat & rate spectra - attachments 10 & 11

adc data item stats - attachment 12

FEE64 temps OK - attachment 13

DSSSD bias & leakage currents OK - attachments 14 & 15

00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17


03:03

analysis of file S505/R5_189 - attachment 18
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20

adc data item stats - attachment 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage currents OK - attachment 23

03:30

per FEE64 1.8.H spectra - attcahments 24 & 25


06:09 no beam 

analysis of file S505/R5_221 - attachment 26
 zero timewarps
 all dead times << 1%

all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28

adc data item stats - attachment 29

FEE64 temps OK - attachment 30

DSSSD bias & leakage currents OK - attachment 31
Attachment 1: R5_154
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260632197 (  880101.4 Hz)
        Other data format:    1287803 (    4348.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1443 (       4.9 Hz)
                               RESUME:       1443 (       4.9 Hz)
                              SYNC100:      33103 (     111.8 Hz)
                              WR48-63:      33103 (     111.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1218711 (    4115.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     391650 (    1322.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      296.139 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               14.978                0.000
  1               49.946                0.000
  2                0.925                0.000
  3                8.130                0.000
  4                0.131               17.535
  5                0.197                0.000
  6                0.760                0.000
  7                0.599                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   32916598      28529          0          0        287        287       4193       4193          0      19569          0      96572
  1   59287487      16678          0          0        710        710       7629       7629          0          0          0      57145
  2   20165095     309168          0          0         52         52       2618       2618          0     303828          0      63600
  3   42178844     305489          0          0        214        214       5322       5322          0     294417          0      32426
  4    5271956       1388          0          0          2          2        692        692          0          0          0      31943
  5   18164680       4584          0          0         28         28       2264       2264          0          0          0      38343
  6   41060613     306981          0          0         61         61       5168       5168          0     296523          0      54824
  7   41586924     314986          0          0         89         89       5217       5217          0     304374          0      16797
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.250s (  723.164 blocks/s,  45.198 Mb/s)
Attachment 2: Screenshot_from_2022-06-24_23-58-03.png
Screenshot_from_2022-06-24_23-58-03.png
Attachment 3: Screenshot_from_2022-06-24_23-56-54.png
Screenshot_from_2022-06-24_23-56-54.png
Attachment 4: Screenshot_from_2022-06-24_23-56-18.png
Screenshot_from_2022-06-24_23-56-18.png
Attachment 5: Screenshot_from_2022-06-24_23-55-53.png
Screenshot_from_2022-06-24_23-55-53.png
Attachment 6: Screenshot_from_2022-06-24_23-55-15.png
Screenshot_from_2022-06-24_23-55-15.png
Attachment 7: Screenshot_from_2022-06-24_23-54-37.png
Screenshot_from_2022-06-24_23-54-37.png
Attachment 8: Screenshot_from_2022-06-24_23-53-22.png
Screenshot_from_2022-06-24_23-53-22.png
Attachment 9: Screenshot_from_2022-06-24_23-51-35.png
Screenshot_from_2022-06-24_23-51-35.png
Attachment 10: Screenshot_from_2022-06-24_23-50-18.png
Screenshot_from_2022-06-24_23-50-18.png
Attachment 11: Screenshot_from_2022-06-24_23-49-08.png
Screenshot_from_2022-06-24_23-49-08.png
Attachment 12: Screenshot_from_2022-06-24_23-48-14.png
Screenshot_from_2022-06-24_23-48-14.png
Attachment 13: Screenshot_from_2022-06-24_23-47-40.png
Screenshot_from_2022-06-24_23-47-40.png
Attachment 14: Screenshot_from_2022-06-24_23-46-10.png
Screenshot_from_2022-06-24_23-46-10.png
Attachment 15: Screenshot_from_2022-06-24_23-44-08.png
Screenshot_from_2022-06-24_23-44-08.png
Attachment 16: Screenshot_from_2022-06-25_00-17-58.png
Screenshot_from_2022-06-25_00-17-58.png
Attachment 17: Screenshot_from_2022-06-25_00-17-36.png
Screenshot_from_2022-06-25_00-17-36.png
Attachment 18: R5_189
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260527466 (  871545.5 Hz)
        Other data format:    1392534 (    4658.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1393 (       4.7 Hz)
                               RESUME:       1393 (       4.7 Hz)
                              SYNC100:      33113 (     110.8 Hz)
                              WR48-63:      33113 (     110.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1323522 (    4427.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     429409 (    1436.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      298.926 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               14.659                0.000
  1               49.890                0.000
  2                0.405                2.804
  3                7.856                0.000
  4                0.000               45.637
  5                0.052                0.000
  6                0.331                0.000
  7                0.206                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   33437852      29241          0          0        298        298       4430       4430          0      19785          0     105011
  1   60873066      16872          0          0        713        713       7723       7723          0          0          0      63589
  2   19991278     336066          0          0         51         51       2502       2502          0     330960          0      70003
  3   42622163     327782          0          0        194        194       5471       5471          0     316452          0      35634
  4    5450705       1394          0          0          0          0        697        697          0          0          0      34276
  5   18430449       4608          0          0         17         17       2287       2287          0          0          0      41850
  6   41735438     335959          0          0         67         67       5311       5311          0     325203          0      60533
  7   37986515     340612          0          0         53         53       4692       4692          0     331122          0      18513
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.010s (  914.031 blocks/s,  57.127 Mb/s)
Attachment 19: Screenshot_from_2022-06-25_03-05-56.png
Screenshot_from_2022-06-25_03-05-56.png
Attachment 20: Screenshot_from_2022-06-25_03-05-35.png
Screenshot_from_2022-06-25_03-05-35.png
Attachment 21: Screenshot_from_2022-06-25_03-04-59.png
Screenshot_from_2022-06-25_03-04-59.png
Attachment 22: Screenshot_from_2022-06-25_03-04-24.png
Screenshot_from_2022-06-25_03-04-24.png
Attachment 23: Screenshot_from_2022-06-25_03-03-54.png
Screenshot_from_2022-06-25_03-03-54.png
Attachment 24: Screenshot_from_2022-06-25_03-31-28.png
Screenshot_from_2022-06-25_03-31-28.png
Attachment 25: Screenshot_from_2022-06-25_03-30-31.png
Screenshot_from_2022-06-25_03-30-31.png
Attachment 26: R5_221
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261819129 (  557711.5 Hz)
        Other data format:     100871 (     214.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        371 (       0.8 Hz)
                               RESUME:        371 (       0.8 Hz)
                              SYNC100:      33749 (      71.9 Hz)
                              WR48-63:      33749 (      71.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      32631 (      69.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      469.453 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.835                0.000
  2                0.000               15.780
  3                0.000                0.000
  4                0.000              132.034
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18255571      37491          0          0          0          0       2430       2430          0      32631          0          1
  1   67878845      18770          0          0        371        371       9014       9014          0          0          0          0
  2    8103831       2048          0          0          0          0       1024       1024          0          0          0          1
  3   41453255      10648          0          0          0          0       5324       5324          0          0          0          0
  4    3384177        880          0          0          0          0        440        440          0          0          0          0
  5   15593071       3880          0          0          0          0       1940       1940          0          0          0          0
  6   55917021      14018          0          0          0          0       7009       7009          0          0          0          0
  7   51233358      13136          0          0          0          0       6568       6568          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.299s (  932.977 blocks/s,  58.311 Mb/s)
Attachment 27: Screenshot_from_2022-06-25_06-11-13.png
Screenshot_from_2022-06-25_06-11-13.png
Attachment 28: Screenshot_from_2022-06-25_06-10-50.png
Screenshot_from_2022-06-25_06-10-50.png
Attachment 29: Screenshot_from_2022-06-25_06-10-11.png
Screenshot_from_2022-06-25_06-10-11.png
Attachment 30: Screenshot_from_2022-06-25_06-09-40.png
Screenshot_from_2022-06-25_06-09-40.png
Attachment 31: Screenshot_from_2022-06-25_06-09-06.png
Screenshot_from_2022-06-25_06-09-06.png
  497   Fri Jun 24 15:04:06 2022 MarcFriday 24th June - evening shift

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

 

 

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

 

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

 

 

Attachment 1: Stats-Screenshot_from_2022-06-24_16-53-47.png
Stats-Screenshot_from_2022-06-24_16-53-47.png
Attachment 2: Temp-Screenshot_from_2022-06-24_16-52-58.png
Temp-Screenshot_from_2022-06-24_16-52-58.png
Attachment 3: HV_LC-Screenshot_from_2022-06-24_16-51-39.png
HV_LC-Screenshot_from_2022-06-24_16-51-39.png
Attachment 4: ucesb-Screenshot_from_2022-06-24_16-57-32.png
ucesb-Screenshot_from_2022-06-24_16-57-32.png
Attachment 5: GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
Attachment 6: Stats-Screenshot_from_2022-06-24_19-21-32.png
Stats-Screenshot_from_2022-06-24_19-21-32.png
Attachment 7: Temp-Screenshot_from_2022-06-24_19-20-48.png
Temp-Screenshot_from_2022-06-24_19-20-48.png
Attachment 8: HV-LC-Screenshot_from_2022-06-24_19-19-43.png
HV-LC-Screenshot_from_2022-06-24_19-19-43.png
Attachment 9: Stats-Screenshot_from_2022-06-24_22-01-09.png
Stats-Screenshot_from_2022-06-24_22-01-09.png
Attachment 10: Temp-Screenshot_from_2022-06-24_22-01-50.png
Temp-Screenshot_from_2022-06-24_22-01-50.png
Attachment 11: HV_LC-Screenshot_from_2022-06-24_21-58-33.png
HV_LC-Screenshot_from_2022-06-24_21-58-33.png
  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
Attachment 1: 220624_0753_stats.png
220624_0753_stats.png
Attachment 2: 220624_0754_Temp.png
220624_0754_Temp.png
Attachment 3: 220624_0755_Bias.png
220624_0755_Bias.png
Attachment 4: R3_359_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260715255 (  897318.9 Hz)
        Other data format:    1204745 (    4146.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1323 (       4.6 Hz)
                               RESUME:       1323 (       4.6 Hz)
                              SYNC100:      33082 (     113.9 Hz)
                              WR48-63:      33082 (     113.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1135935 (    3909.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     353220 (    1215.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.549 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.171                0.000
  1               46.409                0.000
  2                1.006                0.000
  3                4.601                0.000
  4                0.000               37.574
  5                0.031                0.000
  6                0.191                0.000
  7                0.254                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903987      28078          0          0        236        236       4023       4023          0      19560          0      86851
  1   61208192      17252          0          0        718        718       7908       7908          0          0          0      52897
  2   26785987     291397          0          0         72         72       3425       3425          0     284403          0      56573
  3   43119726     289440          0          0        153        153       5319       5319          0     278496          0      29859
  4    4604412       1194          0          0          0          0        597        597          0          0          0      28791
  5   18336996       4656          0          0         16         16       2312       2312          0          0          0      34152
  6   36059875     277258          0          0         42         42       4514       4514          0     268146          0      48935
  7   39696080     295470          0          0         86         86       4984       4984          0     285330          0      15162
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.877s (  917.511 blocks/s,  57.344 Mb/s)
Attachment 5: 220624_0842_Stats_Stop.png
220624_0842_Stats_Stop.png
Attachment 6: 220624_1111_Stats.png
220624_1111_Stats.png
Attachment 7: 220624_1111_Temp.png
220624_1111_Temp.png
Attachment 8: 220624_1112_Bias.png
220624_1112_Bias.png
Attachment 9: 220624_1303_stats.png
220624_1303_stats.png
Attachment 10: 220624_1303_Temp.png
220624_1303_Temp.png
Attachment 11: 220624_1304_Bias.png
220624_1304_Bias.png
Attachment 12: R5_33_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260717099 (  869523.1 Hz)
        Other data format:    1202901 (    4011.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1344 (       4.5 Hz)
                               RESUME:       1344 (       4.5 Hz)
                              SYNC100:      33117 (     110.4 Hz)
                              WR48-63:      33117 (     110.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1133979 (    3782.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354749 (    1183.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      299.839 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.504                0.000
  1               42.102                0.000
  2                0.275                0.000
  3                6.677                0.000
  4                0.000               31.976
  5                0.051                0.000
  6                0.250                0.000
  7                0.412                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31758241      28867          0          0        238        238       4156       4156          0      20079          0      87480
  1   58453105      16104          0          0        635        635       7417       7417          0          0          0      53128
  2   20104679     289945          0          0         41         41       2565       2565          0     284733          0      56849
  3   41709424     287290          0          0        188        188       5262       5262          0     276390          0      29572
  4    5133297       1264          0          0          0          0        632        632          0          0          0      29278
  5   17857346       4470          0          0         22         22       2213       2213          0          0          0      34539
  6   40732502     278221          0          0         59         59       5175       5175          0     267753          0      48740
  7   44968505     296740          0          0        161        161       5697       5697          0     285024          0      15163
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.742s (  921.070 blocks/s,  57.567 Mb/s)
Attachment 13: 220624_1538_Stats.png
220624_1538_Stats.png
Attachment 14: 220624_1538_Temp.png
220624_1538_Temp.png
Attachment 15: 220624_1538_Bias.png
220624_1538_Bias.png
Attachment 16: R5_63_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260796374 (  834020.0 Hz)
        Other data format:    1123626 (    3593.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1191 (       3.8 Hz)
                               RESUME:       1191 (       3.8 Hz)
                              SYNC100:      33165 (     106.1 Hz)
                              WR48-63:      33165 (     106.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1054914 (    3373.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     350507 (    1120.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.698 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.889                0.000
  1               39.498                0.000
  2                0.306                0.000
  3                5.270                0.000
  4                0.000               26.707
  5                0.036                0.000
  6                0.217                0.000
  7                0.175                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903831      29785          0          0        188        188       4074       4074          0      21261          0      87155
  1   61666403      16932          0          0        646        646       7820       7820          0          0          0      55280
  2   20140383     267008          0          0         42         42       2575       2575          0     261774          0      55290
  3   41652352     266495          0          0        164        164       5306       5306          0     255555          0      28879
  4    4871487       1204          0          0          0          0        602        602          0          0          0      28340
  5   17605547       4430          0          0         11         11       2204       2204          0          0          0      33538
  6   41056242     264471          0          0         55         55       5126       5126          0     254109          0      47278
  7   42900129     273301          0          0         85         85       5458       5458          0     262215          0      14747
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.305s (  774.730 blocks/s,  48.421 Mb/s)
  495   Fri Jun 24 00:34:55 2022 AMFriday 24 June 00:00-08:00

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

Attachment 1: 220624_0130_fpgatime.png
220624_0130_fpgatime.png
Attachment 2: 220624_0130_whiterabbit.png
220624_0130_whiterabbit.png
Attachment 3: 220624_0130_temps.png
220624_0130_temps.png
Attachment 4: 220624_0130_stats.png
220624_0130_stats.png
Attachment 5: 220624_0130_bias.png
220624_0130_bias.png
Attachment 6: 220624_0530_fpgatime.png
220624_0530_fpgatime.png
Attachment 7: 220624_0530_whiterabbit.png
220624_0530_whiterabbit.png
Attachment 8: 220624_0530_temps.png
220624_0530_temps.png
Attachment 9: 220624_0530_stats.png
220624_0530_stats.png
Attachment 10: 220624_0530_bias.png
220624_0530_bias.png
Attachment 11: 220624_0330_fpgatime.png
220624_0330_fpgatime.png
Attachment 12: 220624_0330_whiterabbit.png
220624_0330_whiterabbit.png
Attachment 13: 220624_0330_temps.png
220624_0330_temps.png
Attachment 14: 220624_0330_stats.png
220624_0330_stats.png
Attachment 15: 220624_0330_bias.png
220624_0330_bias.png
Attachment 16: 220624_0730_fpga.png
220624_0730_fpga.png
Attachment 17: 220624_0730_whiterabbit.png
220624_0730_whiterabbit.png
Attachment 18: 220624_0730_temps.png
220624_0730_temps.png
Attachment 19: 220624_0730_stats.png
220624_0730_stats.png
Attachment 20: 220624_0730_bias.png
220624_0730_bias.png
  494   Thu Jun 23 15:10:17 2022 Philippos PapadakisThursday 23 June 16:00-24:00
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)
Attachment 1: Temperatures_2022-06-23_16-14-38.png
Temperatures_2022-06-23_16-14-38.png
Attachment 2: stats_2022-06-23_16-16-30.png
stats_2022-06-23_16-16-30.png
Attachment 3: IV2022-06-23_16-17-13.png
IV2022-06-23_16-17-13.png
Attachment 4: Temps_2022-06-23_18-08-04.png
Temps_2022-06-23_18-08-04.png
Attachment 5: Stats_2022-06-23_18-08-38.png
Stats_2022-06-23_18-08-38.png
Attachment 6: IV_2022-06-23_18-09-41.png
IV_2022-06-23_18-09-41.png
Attachment 7: Temps_2022-06-23_20-24-48.png
Temps_2022-06-23_20-24-48.png
Attachment 8: Stats_2022-06-23_20-25-18.png
Stats_2022-06-23_20-25-18.png
Attachment 9: IV2022-06-23_20-25-39.png
IV2022-06-23_20-25-39.png
Attachment 10: Temps_2022-06-23_21-59-26.png
Temps_2022-06-23_21-59-26.png
Attachment 11: Stats_2022-06-23_22-00-01.png
Stats_2022-06-23_22-00-01.png
Attachment 12: IV_2022-06-23_22-00-20.png
IV_2022-06-23_22-00-20.png
Attachment 13: Temps_2022-06-23_23-52-04.png
Temps_2022-06-23_23-52-04.png
Attachment 14: Stats_2022-06-23_23-52-32.png
Stats_2022-06-23_23-52-32.png
Attachment 15: IV_2022-06-23_23-52-50.png
IV_2022-06-23_23-52-50.png
  493   Thu Jun 23 06:56:02 2022 OHThursday 23 June 08:00-16:00
07:56 Taken over from AM
      Current White Rabbit Error status:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51a : 	 28  
      aida08 fault 	 0xf0e9 : 	 0xf132 : 	 73  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA error status:
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link

      Statistics ok - attachment 1
      FEE temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      Analysis of file R3_93 - attachment 4
      Still deadtime in aida02 of ~15%
      Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
      
      Currently 2.2 TB of space on /media/SecondDrive
      Current data rate of 6.23MB/s
      At current data rate this should last 4 days

08:27 Currently no beam
      Problem with Alvarez 3 now
      Tomorrow will have a 1 hour break due to changing of the source

09:36 Beam back

09:59 System wide checks
		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51b : 	 29  
      aida08 fault 	 0xf0e9 : 	 0xf138 : 	 79  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  

      Stats ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7

11:14 FRS group is changing from Hg setting back to Pt

11:36 Back on Pt R3_129

14:14 Stats ok - attachment 8
      Temperature ok - attachment 9
      Bias and leakage currents ok - attachment 10
      System wide checks:
      WR:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc523 : 	 37  
      aida08 fault 	 0xf0e9 : 	 0xf156 : 	 109  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x1b : 	 10  
      aida08 fault 	 0x1a : 	 0x20 : 	 6

      Analysis of R3_156 - attachment 11  
Attachment 1: 220623_0758_Stats.png
220623_0758_Stats.png
Attachment 2: 220623_0759_Temp.png
220623_0759_Temp.png
Attachment 3: 220623_0759_Bias.png
220623_0759_Bias.png
Attachment 4: R3_95_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260243782 (  817307.6 Hz)
        Other data format:    1676218 (    5264.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1230 (       3.9 Hz)
                               RESUME:       1231 (       3.9 Hz)
                              SYNC100:      33186 (     104.2 Hz)
                              WR48-63:      33186 (     104.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1607385 (    5048.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440709 (    1384.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.416 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.258                0.000
  1               47.390                0.000
  2                1.040                0.000
  3                6.094                0.000
  4                0.003               59.113
  5                0.094                0.000
  6                0.304                0.000
  7                0.013                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20778195      27751          0          0         46         46       2749       2749          0      22161          0      72140
  1   62611625      17407          0          0        705        706       7998       7998          0          0          0      66280
  2   31916579     410233          0          0         83         83       4122       4122          0     401823          0      94767
  3   52868439     408659          0          0        312        312       6874       6874          0     394287          0      28791
  4    5535248       1334          0          0          1          1        666        666          0          0          0      39988
  5   19704728       4912          0          0         25         25       2431       2431          0          0          0      52543
  6   35689179     392725          0          0         47         47       4446       4446          0     383739          0      67737
  7   31139789     413197          0          0         11         11       3900       3900          0     405375          0      18463
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.666s (  923.094 blocks/s,  57.693 Mb/s)
Attachment 5: 220623_0958_Stats.png
220623_0958_Stats.png
Attachment 6: 220623_0959_Temp.png
220623_0959_Temp.png
Attachment 7: 220623_1000_Bias.png
220623_1000_Bias.png
Attachment 8: 220623_1413_Stats.png
220623_1413_Stats.png
Attachment 9: 220623_1413_Temp.png
220623_1413_Temp.png
Attachment 10: 220623_1414_bias.png
220623_1414_bias.png
Attachment 11: R3_156_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260453729 (  852518.1 Hz)
        Other data format:    1466271 (    4799.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1362 (       4.5 Hz)
                               RESUME:       1362 (       4.5 Hz)
                              SYNC100:      33138 (     108.5 Hz)
                              WR48-63:      33138 (     108.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1397271 (    4573.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     435545 (    1425.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      305.511 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.365                0.000
  1               54.806                0.000
  2                0.562                0.000
  3                5.946                0.000
  4                0.000               35.352
  5                0.068                0.000
  6                0.334                0.000
  7                0.054                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30150644      28922          0          0        254        254       3938       3938          0      20538          0     108754
  1   64977250      18108          0          0        757        757       8297       8297          0          0          0      62895
  2   28023912     358551          0          0         83         83       3523       3523          0     351339          0      69361
  3   44222269     352853          0          0        180        180       5536       5536          0     341421          0      37017
  4    5005333       1272          0          0          0          0        636        636          0          0          0      36369
  5   19466878       5006          0          0         20         20       2483       2483          0          0          0      42187
  6   36422600     339861          0          0         37         37       4700       4700          0     330387          0      59710
  7   32184843     361698          0          0         31         31       4025       4025          0     353586          0      19252
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.418s (  903.496 blocks/s,  56.469 Mb/s)
  492   Wed Jun 22 23:18:45 2022 AM, OH, TD, MAThursday 23 June
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
      Attempted to recover restarting merger but this caused 01 to drop out.

00:10 DAQ recovered and AIDA01 recovered with a reboot


01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good

03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good

05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
Attachment 1: 220623_0100_whiterabbit.png
220623_0100_whiterabbit.png
Attachment 2: 220623_0100_temps.png
220623_0100_temps.png
Attachment 3: 220623_0100_stats.png
220623_0100_stats.png
Attachment 4: 220623_0100_bias.png
220623_0100_bias.png
Attachment 5: 220623_0300_whiterabbit.png
220623_0300_whiterabbit.png
Attachment 6: 220623_0300_fpgatime.png
220623_0300_fpgatime.png
Attachment 7: 220623_0300_temps.png
220623_0300_temps.png
Attachment 8: 220623_0300_stats.png
220623_0300_stats.png
Attachment 9: 220623_0300_bias.png
220623_0300_bias.png
Attachment 10: 220623_0500_fpgatime.png
220623_0500_fpgatime.png
Attachment 11: 220623_0500_whiterabbit.png
220623_0500_whiterabbit.png
Attachment 12: 220623_0500_temps.png
220623_0500_temps.png
Attachment 13: 220623_0500_stats.png
220623_0500_stats.png
Attachment 14: 220623_0500_bias.png
220623_0500_bias.png
Attachment 15: 220623_0700_fpgatime.png
220623_0700_fpgatime.png
Attachment 16: 220623_0700_whiterabbit.png
220623_0700_whiterabbit.png
Attachment 17: 220623_0700_temps.png
220623_0700_temps.png
Attachment 18: 220623_0700_stats.png
220623_0700_stats.png
Attachment 19: 220623_0700_bias.png
220623_0700_bias.png
  491   Wed Jun 22 15:09:37 2022 MAWednesday 22 June 16:00-00:00

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

 

Attachment 1: BiasCurrent2022-06-22_17-11-07.png
BiasCurrent2022-06-22_17-11-07.png
Attachment 2: Statistics_2022-06-22_17-13-52.png
Statistics_2022-06-22_17-13-52.png
Attachment 3: Temperature2022-06-22_17-12-35.png
Temperature2022-06-22_17-12-35.png
Attachment 4: BiasCurrent2022-06-22_19-00-31.png
BiasCurrent2022-06-22_19-00-31.png
Attachment 5: Temperature2022-06-22_19-02-18.png
Temperature2022-06-22_19-02-18.png
Attachment 6: Statics2022-06-22_19-03-11.png
Statics2022-06-22_19-03-11.png
Attachment 7: BiasCurrent2022-06-22_21-04-38.png
BiasCurrent2022-06-22_21-04-38.png
Attachment 8: Statistics2022-06-22_21-07-07.png
Statistics2022-06-22_21-07-07.png
Attachment 9: Temperature2022-06-22_21-05-51.png
Temperature2022-06-22_21-05-51.png
Attachment 10: BiasCurrent2022-06-22_23-29-40.png
BiasCurrent2022-06-22_23-29-40.png
Attachment 11: Statistics2022-06-22_23-31-30.png
Statistics2022-06-22_23-31-30.png
Attachment 12: Temperature2022-06-22_23-30-28.png
Temperature2022-06-22_23-30-28.png
  490   Wed Jun 22 10:09:12 2022 PJCSINFO: FEE64 supply Voltages

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

  489   Wed Jun 22 07:12:46 2022 MSWednesday 22 June 8:00-16:00

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

 

Attachment 1: Screenshot_from_2022-06-22_08-11-30.png
Screenshot_from_2022-06-22_08-11-30.png
Attachment 2: Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-06-22_10-00-13.png
Screenshot_from_2022-06-22_10-00-13.png
Attachment 7: Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_from_2022-06-22_11-59-55.png
Screenshot_from_2022-06-22_11-59-55.png
Attachment 10: R2_104_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260553751 (  867725.3 Hz)
        Other data format:    1366249 (    4550.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1464 (       4.9 Hz)
                               RESUME:       1464 (       4.9 Hz)
                              SYNC100:      33119 (     110.3 Hz)
                              WR48-63:      33119 (     110.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1297083 (    4319.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     518974 (    1728.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      300.272 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.776                0.000
  1               47.563                0.000
  2                0.466                0.000
  3                4.438                0.000
  4                0.149               14.780
  5               16.612                0.000
  6                0.972                0.000
  7                0.380                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27837591      27876          0          0        187        187       3557       3557          0      20388          0     104437
  1   57538232      16015          0          0        651        652       7356       7356          0          0          0      64114
  2   23047755     331545          0          0         63         63       2949       2949          0     325521          0      64002
  3   44324924     329582          0          0        146        145       5671       5671          0     317949          0      37242
  4   11203550       2766          0          0         57         57       1326       1326          0          0          0      90620
  5   28839527       7636          0          0        219        219       3599       3599          0          0          0      65648
  6   37663072     317251          0          0         87         87       4772       4772          0     307533          0      63360
  7   30099100     333578          0          0         54         54       3889       3889          0     325692          0      29551
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.320s (  932.392 blocks/s,  58.275 Mb/s)
Attachment 11: Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-06-22_14-03-14.png
Screenshot_from_2022-06-22_14-03-14.png
Attachment 14: Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_from_2022-06-22_15-58-16.png
Screenshot_from_2022-06-22_15-58-16.png
  488   Tue Jun 21 23:29:16 2022 TDWednesday 22 June 00:00-08:00
00.28 all histograms & stats zero'd
      baseline counters

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

BNC PB-5 pulser
 to p+n FEE64s only
  amplitude 1.0V
  attenuator x1
  decay time 1ms
  polarity +
  frequency 2Hz

all AD9252 waveform ADCs disabled

Tape Server -> no storage mode

00.36 check ASIC control all FEE64s all ASICs

      all system wide checks OK

00.41 BNC PB-5 pulser rate increased from 2Hz to 22Hz per request by HA/NH - to ensure DTAS correlation checker has sufficient rate of identified pulser events 

per p+n FEE64 1.8.L spectra - attachment 1
 aida01 pulser peak width 88 ch FWHM

per FEE64 rate spectra - attachment 2

adc, pause and correlation scaler data items - push, aida01 and aida02 stats - attachments 3-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage current OK - attachments 10 & 11

02.28 aida05 zero adc data items, unable to halt aida05 - all other FEE64s OK

      restart NewMerger -> now able to halt aida05

      DAQ go OK, merge rate c. 1.4M data items/s

per FEE64 1.8.H spectra - attachments 12 & 13
 approx flat distribution to c. 10GeV


adc data items stats OK - attachments 14

FEE64 temps OK - attachment 15

DSSSD bias & leakage current OK - attachments 16 & 17

02.55 Some problems with GO4 data analysis for PID (FRS PID looks OK). 204Pt setting. Will commence writing data to disk file S505/R2_0

04.12 per FEE64 1.8.H spectra - attachments 18 & 19
      c. 9GeV peak in upstream DSSSDs (aida01, 2, 3 & 4) indicating (high energy) heavy fragments are stopping
      Low energy events near c. 200MeV lower mass, high energy, low energy loss - fission fragments?

04.20 per FEE64 stats spectra - attachment 20
      indicates implant x-y distribution - off centre (more significantly in x than in y) but DSSSDs appear to see most of distribution
      remember raw distro will have significant fraction of fission fragments - may be misleading - need to gate on high energy events to see x-y distro of ions of 
interest

adc data items stats OK - attachments 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage current OK - attachments 23



06.10 per FEE64 rate spectra - attachment 24

per FEE64 1.8.H spectra - attachments 25 & 26

per FEE64 stats spectra - attachment 27
     
adc data items stats OK - attachments 28

FEE64 temps OK - attachment 29

system wide checks OK *except* WR & FPGA errors - see attachments 30 & 31

DSSSD bias & leakage current OK - attachments 32

07.52 analysis of file S505/R2_47 - attachment 33
      all FEE64 dead times < 1% *except* aida02 c. 7%
Attachment 1: Screenshot_from_2022-06-22_00-50-20.png
Screenshot_from_2022-06-22_00-50-20.png
Attachment 2: Screenshot_from_2022-06-22_00-49-07.png
Screenshot_from_2022-06-22_00-49-07.png
Attachment 3: Screenshot_from_2022-06-22_00-47-56.png
Screenshot_from_2022-06-22_00-47-56.png
Attachment 4: Screenshot_from_2022-06-22_00-46-57.png
Screenshot_from_2022-06-22_00-46-57.png
Attachment 5: Screenshot_from_2022-06-22_00-46-17.png
Screenshot_from_2022-06-22_00-46-17.png
Attachment 6: Screenshot_from_2022-06-22_00-45-33.png
Screenshot_from_2022-06-22_00-45-33.png
Attachment 7: Screenshot_from_2022-06-22_00-44-58.png
Screenshot_from_2022-06-22_00-44-58.png
Attachment 8: Screenshot_from_2022-06-22_00-44-13.png
Screenshot_from_2022-06-22_00-44-13.png
Attachment 9: Screenshot_from_2022-06-22_00-36-06.png
Screenshot_from_2022-06-22_00-36-06.png
Attachment 10: Screenshot_from_2022-06-22_00-35-17.png
Screenshot_from_2022-06-22_00-35-17.png
Attachment 11: Screenshot_from_2022-06-22_00-34-01.png
Screenshot_from_2022-06-22_00-34-01.png
Attachment 12: Screenshot_from_2022-06-22_02-44-02.png
Screenshot_from_2022-06-22_02-44-02.png
Attachment 13: Screenshot_from_2022-06-22_02-41-13.png
Screenshot_from_2022-06-22_02-41-13.png
Attachment 14: Screenshot_from_2022-06-22_02-40-14.png
Screenshot_from_2022-06-22_02-40-14.png
Attachment 15: Screenshot_from_2022-06-22_02-39-42.png
Screenshot_from_2022-06-22_02-39-42.png
Attachment 16: Screenshot_from_2022-06-22_02-39-01.png
Screenshot_from_2022-06-22_02-39-01.png
Attachment 17: Screenshot_from_2022-06-22_02-38-24.png
Screenshot_from_2022-06-22_02-38-24.png
Attachment 18: Screenshot_from_2022-06-22_04-10-47.png
Screenshot_from_2022-06-22_04-10-47.png
Attachment 19: Screenshot_from_2022-06-22_04-10-00.png
Screenshot_from_2022-06-22_04-10-00.png
Attachment 20: Screenshot_from_2022-06-22_04-23-13.png
Screenshot_from_2022-06-22_04-23-13.png
Attachment 21: Screenshot_from_2022-06-22_04-20-06.png
Screenshot_from_2022-06-22_04-20-06.png
Attachment 22: Screenshot_from_2022-06-22_04-19-34.png
Screenshot_from_2022-06-22_04-19-34.png
Attachment 23: Screenshot_from_2022-06-22_04-19-00.png
Screenshot_from_2022-06-22_04-19-00.png
Attachment 24: Screenshot_from_2022-06-22_06-10-41.png
Screenshot_from_2022-06-22_06-10-41.png
Attachment 25: Screenshot_from_2022-06-22_06-09-33.png
Screenshot_from_2022-06-22_06-09-33.png
Attachment 26: Screenshot_from_2022-06-22_06-08-08.png
Screenshot_from_2022-06-22_06-08-08.png
Attachment 27: Screenshot_from_2022-06-22_06-06-50.png
Screenshot_from_2022-06-22_06-06-50.png
Attachment 28: Screenshot_from_2022-06-22_06-05-58.png
Screenshot_from_2022-06-22_06-05-58.png
Attachment 29: Screenshot_from_2022-06-22_06-05-20.png
Screenshot_from_2022-06-22_06-05-20.png
Attachment 30: Screenshot_from_2022-06-22_06-04-27.png
Screenshot_from_2022-06-22_06-04-27.png
Attachment 31: Screenshot_from_2022-06-22_06-03-53.png
Screenshot_from_2022-06-22_06-03-53.png
Attachment 32: Screenshot_from_2022-06-22_06-03-05.png
Screenshot_from_2022-06-22_06-03-05.png
Attachment 33: R2_47
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259428432 (  714533.4 Hz)
        Other data format:    2491568 (    6862.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        881 (       2.4 Hz)
                               RESUME:        880 (       2.4 Hz)
                              SYNC100:      33352 (      91.9 Hz)
                              WR48-63:      33352 (      91.9 Hz)
                           FEE64 disc:    1422279 (    3917.3 Hz)
                             MBS info:    1000824 (    2756.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     416179 (    1146.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      363.074 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.857                0.000
  1               26.452                0.000
  2                0.132                0.000
  3                1.570                0.000
  4                0.002               46.041
  5                0.661                0.000
  6                0.312                0.000
  7                0.027                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   25701314     249377          0          0         56         56       3411       3411     217219      25224          0      83012
  1   63351849     247459          0          0        567        566       8361       8361     229604          0          0      62244
  2   23166597     387869          0          0         37         37       3010       3010     134065     247710          0      48973
  3   48876539     363237          0          0        105        105       6171       6171     104940     245745          0      28921
  4    8006429     267186          0          0          3          3       1045       1045     265090          0          0      68788
  5   22095663     170195          0          0         36         36       2789       2789     164545          0          0      52757
  6   40483353     472437          0          0         66         66       5106       5106     227985     234108          0      48544
  7   27746688     333808          0          0         11         11       3459       3459      78831     248037          0      22940
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.072s (  912.402 blocks/s,  57.025 Mb/s)
  487   Tue Jun 21 10:19:05 2022 OH MLTuesday 21st June 08:00-00:00
11:19 Stats ok 0xa on all FEE - attachment 1
      Current pulser settings - attachment 2
      FEE Temperatures ok - attachment 3
      Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
      Layout 1 after recovering channels - attachment 4
      New stats (Note increased rate on 3 and 7) attachment 5
      Bias and leakage currents ok - attachment 6

      S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.

14:05 Have had it confirmed that the flange beam dump is in position in S4

15:43 Stats ok -attachment 7
      Temp ok - attachment 8
      Bias and Leakage current ok - attachment 9


17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
      As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
      During the experiment this will be changed to SC1 trigger to solve this.

19:49: Stats ok - attachment 10
       temperatures ok - attachment 11
       Bias - Leakage current ok - attachment 12
Attachment 1: 220621_1119_Stats.png
220621_1119_Stats.png
Attachment 2: 220621_1120_Pulser.png
220621_1120_Pulser.png
Attachment 3: 220621_1120_Temp.png
220621_1120_Temp.png
Attachment 4: 220621_1123_Layout1.png
220621_1123_Layout1.png
Attachment 5: 220621_1123_Stats.png
220621_1123_Stats.png
Attachment 6: 220621_1123_Bias.png
220621_1123_Bias.png
Attachment 7: 220621_1542_Stats.png
220621_1542_Stats.png
Attachment 8: 220621_1542_Temp.png
220621_1542_Temp.png
Attachment 9: 220621_1543_Bias.png
220621_1543_Bias.png
Attachment 10: Stats-Screenshot_from_2022-06-21_19-50-19.png
Stats-Screenshot_from_2022-06-21_19-50-19.png
Attachment 11: Temp-Screenshot_from_2022-06-21_19-48-56.png
Temp-Screenshot_from_2022-06-21_19-48-56.png
Attachment 12: HV_LC-Screenshot_from_2022-06-21_19-47-49.png
HV_LC-Screenshot_from_2022-06-21_19-47-49.png
  486   Mon Jun 20 09:26:23 2022 OH, TDMonday 20 June
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11
Attachment 1: 220620_1116_stats_no_waveform.png
220620_1116_stats_no_waveform.png
Attachment 2: 220620_1223_stats_no_waveform.png
220620_1223_stats_no_waveform.png
Attachment 3: Screenshot_from_2022-06-20_21-17-41.png
Screenshot_from_2022-06-20_21-17-41.png
Attachment 4: Screenshot_from_2022-06-20_21-14-12.png
Screenshot_from_2022-06-20_21-14-12.png
Attachment 5: Screenshot_from_2022-06-20_21-12-06.png
Screenshot_from_2022-06-20_21-12-06.png
Attachment 6: Screenshot_from_2022-06-20_21-10-55.png
Screenshot_from_2022-06-20_21-10-55.png
Attachment 7: Screenshot_from_2022-06-20_21-08-58.png
Screenshot_from_2022-06-20_21-08-58.png
Attachment 8: Screenshot_from_2022-06-20_21-04-58.png
Screenshot_from_2022-06-20_21-04-58.png
Attachment 9: Screenshot_from_2022-06-20_21-04-00.png
Screenshot_from_2022-06-20_21-04-00.png
Attachment 10: Screenshot_from_2022-06-20_21-02-07.png
Screenshot_from_2022-06-20_21-02-07.png
Attachment 11: Screenshot_from_2022-06-20_21-01-07.png
Screenshot_from_2022-06-20_21-01-07.png
  485   Sun Jun 19 19:55:20 2022 NHaida03 and aida04
aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour
Attachment 1: Screenshot_2022-06-19_205528.png
Screenshot_2022-06-19_205528.png
Attachment 2: Screenshot_2022-06-19_205544.png
Screenshot_2022-06-19_205544.png
  484   Sun Jun 19 12:36:10 2022 OHSunday 19th June
Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9
Attachment 1: 1.jpg
1.jpg
Attachment 2: 2.jpg
2.jpg
Attachment 3: 3.jpg
3.jpg
Attachment 4: 4.jpg
4.jpg
Attachment 5: 220618_1753_stats_no_waveform.png
220618_1753_stats_no_waveform.png
  483   Sun Jun 19 00:22:24 2022 TDSunday 19 June
01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM
Attachment 1: Screenshot_from_2022-06-19_01-24-50.png
Screenshot_from_2022-06-19_01-24-50.png
Attachment 2: Screenshot_from_2022-06-19_01-24-24.png
Screenshot_from_2022-06-19_01-24-24.png
Attachment 3: Screenshot_from_2022-06-19_01-20-50.png
Screenshot_from_2022-06-19_01-20-50.png
Attachment 4: Screenshot_from_2022-06-19_01-20-09.png
Screenshot_from_2022-06-19_01-20-09.png
Attachment 5: Screenshot_from_2022-06-19_01-19-37.png
Screenshot_from_2022-06-19_01-19-37.png
Attachment 6: Screenshot_from_2022-06-19_01-18-35.png
Screenshot_from_2022-06-19_01-18-35.png
Attachment 7: Screenshot_from_2022-06-19_01-18-05.png
Screenshot_from_2022-06-19_01-18-05.png
Attachment 8: Screenshot_from_2022-06-19_01-17-28.png
Screenshot_from_2022-06-19_01-17-28.png
Attachment 9: Screenshot_from_2022-06-19_01-15-44.png
Screenshot_from_2022-06-19_01-15-44.png
Attachment 10: Screenshot_from_2022-06-19_01-14-59.png
Screenshot_from_2022-06-19_01-14-59.png
Attachment 11: Screenshot_from_2022-06-19_01-14-24.png
Screenshot_from_2022-06-19_01-14-24.png
Attachment 12: Screenshot_from_2022-06-19_01-13-43.png
Screenshot_from_2022-06-19_01-13-43.png
Attachment 13: Screenshot_from_2022-06-19_01-13-03.png
Screenshot_from_2022-06-19_01-13-03.png
Attachment 14: Screenshot_from_2022-06-19_01-12-16.png
Screenshot_from_2022-06-19_01-12-16.png
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
Attachment 1: 220617_1322_stats.png
220617_1322_stats.png
Attachment 2: 220617_1420_stats.png
220617_1420_stats.png
Attachment 3: 220617_11428_stats.png
220617_11428_stats.png
Attachment 4: 220617_1428_layout1.png
220617_1428_layout1.png
Attachment 5: 220617_1440_stats_waveform_on.png
220617_1440_stats_waveform_on.png
Attachment 6: 220617_1444_stats_no_waveform.png
220617_1444_stats_no_waveform.png
Attachment 7: 220617_1459_stats_waveform_on.png
220617_1459_stats_waveform_on.png
Attachment 8: 220617_1517_stats_no_waveform.png
220617_1517_stats_no_waveform.png
Attachment 9: 220617_1624_stats_waveform_on.png
220617_1624_stats_waveform_on.png
Attachment 10: 220617_1626_stats_waveform_off.png
220617_1626_stats_waveform_off.png
Attachment 11: 220617_1649_stats_waveform_on.png
220617_1649_stats_waveform_on.png
Attachment 12: 220617_1652_no_waveform.png
220617_1652_no_waveform.png
Attachment 13: 220617_1709_waveform_on.png
220617_1709_waveform_on.png
Attachment 14: 220617_1724_stats_no_waveform.png
220617_1724_stats_no_waveform.png
Attachment 15: 220617_1729_stats_waveform_on.png
220617_1729_stats_waveform_on.png
Attachment 16: 220617_1733_stats_no_waveform.png
220617_1733_stats_no_waveform.png
Attachment 17: 220617_1736_0x21.png
220617_1736_0x21.png
Attachment 18: 220617_1738_0x14.png
220617_1738_0x14.png
Attachment 19: 220617_1738_0x12.png
220617_1738_0x12.png
Attachment 20: 220617_1807_stats_waveform_on.png
220617_1807_stats_waveform_on.png
Attachment 21: 220617_1832_stats_no_waveform.png
220617_1832_stats_no_waveform.png
Attachment 22: 220617_1844_0xf.png
220617_1844_0xf.png
Attachment 23: 220617_1845_stats_waveform_on.png
220617_1845_stats_waveform_on.png
  481   Tue Jun 14 12:15:33 2022 NHS450 Archived
All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 
Attachment 1: s450_aida_tape.txt
R1_0.gz: (0x00000009) exists archived, archive_id:13
R1_100.gz: (0x00000009) exists archived, archive_id:13
R1_101.gz: (0x00000009) exists archived, archive_id:13
R1_102.gz: (0x00000009) exists archived, archive_id:13
R1_103.gz: (0x00000009) exists archived, archive_id:13
R1_104.gz: (0x00000009) exists archived, archive_id:13
R1_105.gz: (0x00000009) exists archived, archive_id:13
R1_106.gz: (0x00000009) exists archived, archive_id:13
R1_107.gz: (0x00000009) exists archived, archive_id:13
R1_108.gz: (0x00000009) exists archived, archive_id:13
R1_109.gz: (0x00000009) exists archived, archive_id:13
R1_10.gz: (0x00000009) exists archived, archive_id:13
R1_110.gz: (0x00000009) exists archived, archive_id:13
R1_111.gz: (0x00000009) exists archived, archive_id:13
R1_112.gz: (0x00000009) exists archived, archive_id:13
R1_113.gz: (0x00000009) exists archived, archive_id:13
R1_114.gz: (0x00000009) exists archived, archive_id:13
R1_115.gz: (0x00000009) exists archived, archive_id:13
R1_116.gz: (0x00000009) exists archived, archive_id:13
R1_117.gz: (0x00000009) exists archived, archive_id:13
R1_118.gz: (0x00000009) exists archived, archive_id:13
R1_119.gz: (0x00000009) exists archived, archive_id:13
R1_11.gz: (0x00000009) exists archived, archive_id:13
R1_120.gz: (0x00000009) exists archived, archive_id:13
R1_121.gz: (0x00000009) exists archived, archive_id:13
R1_122.gz: (0x00000009) exists archived, archive_id:13
R1_123.gz: (0x00000009) exists archived, archive_id:13
R1_124.gz: (0x00000009) exists archived, archive_id:13
R1_125.gz: (0x00000009) exists archived, archive_id:13
R1_126.gz: (0x00000009) exists archived, archive_id:13
R1_127.gz: (0x00000009) exists archived, archive_id:13
R1_128.gz: (0x00000009) exists archived, archive_id:13
R1_129.gz: (0x00000009) exists archived, archive_id:13
R1_12.gz: (0x00000009) exists archived, archive_id:13
R1_130.gz: (0x00000009) exists archived, archive_id:13
R1_131.gz: (0x00000009) exists archived, archive_id:13
R1_132.gz: (0x00000009) exists archived, archive_id:13
R1_133.gz: (0x00000009) exists archived, archive_id:13
R1_134.gz: (0x00000009) exists archived, archive_id:13
R1_135.gz: (0x00000009) exists archived, archive_id:13
R1_136.gz: (0x00000009) exists archived, archive_id:13
R1_137.gz: (0x00000009) exists archived, archive_id:13
R1_138.gz: (0x00000009) exists archived, archive_id:13
R1_139.gz: (0x00000009) exists archived, archive_id:13
R1_13.gz: (0x00000009) exists archived, archive_id:13
R1_140.gz: (0x00000009) exists archived, archive_id:13
R1_141.gz: (0x00000009) exists archived, archive_id:13
R1_142.gz: (0x00000009) exists archived, archive_id:13
R1_143.gz: (0x00000009) exists archived, archive_id:13
R1_144.gz: (0x00000009) exists archived, archive_id:13
R1_145.gz: (0x00000009) exists archived, archive_id:13
R1_146.gz: (0x00000009) exists archived, archive_id:13
R1_147.gz: (0x00000009) exists archived, archive_id:13
R1_148.gz: (0x00000009) exists archived, archive_id:13
R1_149.gz: (0x00000009) exists archived, archive_id:13
R1_14.gz: (0x00000009) exists archived, archive_id:13
R1_150.gz: (0x00000009) exists archived, archive_id:13
R1_151.gz: (0x00000009) exists archived, archive_id:13
R1_152.gz: (0x00000009) exists archived, archive_id:13
R1_153.gz: (0x00000009) exists archived, archive_id:13
R1_154.gz: (0x00000009) exists archived, archive_id:13
R1_155.gz: (0x00000009) exists archived, archive_id:13
R1_156.gz: (0x00000009) exists archived, archive_id:13
R1_157.gz: (0x00000009) exists archived, archive_id:13
R1_158.gz: (0x00000009) exists archived, archive_id:13
R1_159.gz: (0x00000009) exists archived, archive_id:13
R1_15.gz: (0x00000009) exists archived, archive_id:13
R1_160.gz: (0x00000009) exists archived, archive_id:13
R1_161.gz: (0x00000009) exists archived, archive_id:13
R1_162.gz: (0x00000009) exists archived, archive_id:13
R1_163.gz: (0x00000009) exists archived, archive_id:13
R1_164.gz: (0x00000009) exists archived, archive_id:13
R1_165.gz: (0x00000009) exists archived, archive_id:13
R1_166.gz: (0x00000009) exists archived, archive_id:13
R1_167.gz: (0x00000009) exists archived, archive_id:13
R1_168.gz: (0x00000009) exists archived, archive_id:13
R1_169.gz: (0x00000009) exists archived, archive_id:13
R1_16.gz: (0x00000009) exists archived, archive_id:13
R1_170.gz: (0x00000009) exists archived, archive_id:13
R1_171.gz: (0x00000009) exists archived, archive_id:13
R1_172.gz: (0x00000009) exists archived, archive_id:13
R1_173.gz: (0x00000009) exists archived, archive_id:13
R1_174.gz: (0x00000009) exists archived, archive_id:13
R1_175.gz: (0x00000009) exists archived, archive_id:13
R1_176.gz: (0x00000009) exists archived, archive_id:13
R1_177.gz: (0x00000009) exists archived, archive_id:13
R1_178.gz: (0x00000009) exists archived, archive_id:13
R1_179.gz: (0x00000009) exists archived, archive_id:13
R1_17.gz: (0x00000009) exists archived, archive_id:13
R1_180.gz: (0x00000009) exists archived, archive_id:13
R1_181.gz: (0x00000009) exists archived, archive_id:13
R1_182.gz: (0x00000009) exists archived, archive_id:13
R1_183.gz: (0x00000009) exists archived, archive_id:13
R1_184.gz: (0x00000009) exists archived, archive_id:13
R1_185.gz: (0x00000009) exists archived, archive_id:13
R1_186.gz: (0x00000009) exists archived, archive_id:13
R1_187.gz: (0x00000009) exists archived, archive_id:13
R1_188.gz: (0x00000009) exists archived, archive_id:13
R1_189.gz: (0x00000009) exists archived, archive_id:13
R1_18.gz: (0x00000009) exists archived, archive_id:13
R1_190.gz: (0x00000009) exists archived, archive_id:13
R1_191.gz: (0x00000009) exists archived, archive_id:13
R1_192.gz: (0x00000009) exists archived, archive_id:13
R1_193.gz: (0x00000009) exists archived, archive_id:13
R1_194.gz: (0x00000009) exists archived, archive_id:13
R1_195.gz: (0x00000009) exists archived, archive_id:13
R1_196.gz: (0x00000009) exists archived, archive_id:13
R1_197.gz: (0x00000009) exists archived, archive_id:13
R1_198.gz: (0x00000009) exists archived, archive_id:13
R1_199.gz: (0x00000009) exists archived, archive_id:13
R1_19.gz: (0x00000009) exists archived, archive_id:13
R1_1.gz: (0x00000009) exists archived, archive_id:13
R1_200.gz: (0x00000009) exists archived, archive_id:13
R1_201.gz: (0x00000009) exists archived, archive_id:13
R1_202.gz: (0x00000009) exists archived, archive_id:13
R1_203.gz: (0x00000009) exists archived, archive_id:13
R1_204.gz: (0x00000009) exists archived, archive_id:13
R1_205.gz: (0x00000009) exists archived, archive_id:13
R1_206.gz: (0x00000009) exists archived, archive_id:13
R1_207.gz: (0x00000009) exists archived, archive_id:13
R1_208.gz: (0x00000009) exists archived, archive_id:13
R1_209.gz: (0x00000009) exists archived, archive_id:13
R1_20.gz: (0x00000009) exists archived, archive_id:13
R1_210.gz: (0x00000009) exists archived, archive_id:13
R1_211.gz: (0x00000009) exists archived, archive_id:13
R1_212.gz: (0x00000009) exists archived, archive_id:13
R1_213.gz: (0x00000009) exists archived, archive_id:13
R1_214.gz: (0x00000009) exists archived, archive_id:13
R1_215.gz: (0x00000009) exists archived, archive_id:13
R1_216.gz: (0x00000009) exists archived, archive_id:13
R1_217.gz: (0x00000009) exists archived, archive_id:13
R1_218.gz: (0x00000009) exists archived, archive_id:13
R1_219.gz: (0x00000009) exists archived, archive_id:13
R1_21.gz: (0x00000009) exists archived, archive_id:13
R1_220.gz: (0x00000009) exists archived, archive_id:13
R1_221.gz: (0x00000009) exists archived, archive_id:13
R1_222.gz: (0x00000009) exists archived, archive_id:13
R1_223.gz: (0x00000009) exists archived, archive_id:13
R1_224.gz: (0x00000009) exists archived, archive_id:13
R1_225.gz: (0x00000009) exists archived, archive_id:13
R1_226.gz: (0x00000009) exists archived, archive_id:13
R1_227.gz: (0x00000009) exists archived, archive_id:13
R1_228.gz: (0x00000009) exists archived, archive_id:13
R1_229.gz: (0x00000009) exists archived, archive_id:13
R1_22.gz: (0x00000009) exists archived, archive_id:13
R1_230.gz: (0x00000009) exists archived, archive_id:13
R1_231.gz: (0x00000009) exists archived, archive_id:13
R1_232.gz: (0x00000009) exists archived, archive_id:13
R1_233.gz: (0x00000009) exists archived, archive_id:13
R1_234.gz: (0x00000009) exists archived, archive_id:13
R1_235.gz: (0x00000009) exists archived, archive_id:13
R1_236.gz: (0x00000009) exists archived, archive_id:13
R1_237.gz: (0x00000009) exists archived, archive_id:13
R1_238.gz: (0x00000009) exists archived, archive_id:13
R1_239.gz: (0x00000009) exists archived, archive_id:13
R1_23.gz: (0x00000009) exists archived, archive_id:13
R1_240.gz: (0x00000009) exists archived, archive_id:13
R1_241.gz: (0x00000009) exists archived, archive_id:13
R1_242.gz: (0x00000009) exists archived, archive_id:13
R1_243.gz: (0x00000009) exists archived, archive_id:13
R1_244.gz: (0x00000009) exists archived, archive_id:13
R1_245.gz: (0x00000009) exists archived, archive_id:13
R1_246.gz: (0x00000009) exists archived, archive_id:13
R1_247.gz: (0x00000009) exists archived, archive_id:13
R1_248.gz: (0x00000009) exists archived, archive_id:13
R1_249.gz: (0x00000009) exists archived, archive_id:13
R1_24.gz: (0x00000009) exists archived, archive_id:13
R1_250.gz: (0x00000009) exists archived, archive_id:13
R1_251.gz: (0x00000009) exists archived, archive_id:13
R1_252.gz: (0x00000009) exists archived, archive_id:13
R1_253.gz: (0x00000009) exists archived, archive_id:13
R1_254.gz: (0x00000009) exists archived, archive_id:13
R1_255.gz: (0x00000009) exists archived, archive_id:13
R1_256.gz: (0x00000009) exists archived, archive_id:13
R1_257.gz: (0x00000009) exists archived, archive_id:13
R1_258.gz: (0x00000009) exists archived, archive_id:13
R1_259.gz: (0x00000009) exists archived, archive_id:13
R1_25.gz: (0x00000009) exists archived, archive_id:13
R1_260.gz: (0x00000009) exists archived, archive_id:13
R1_261.gz: (0x00000009) exists archived, archive_id:13
R1_262.gz: (0x00000009) exists archived, archive_id:13
R1_263.gz: (0x00000009) exists archived, archive_id:13
R1_264.gz: (0x00000009) exists archived, archive_id:13
R1_265.gz: (0x00000009) exists archived, archive_id:13
R1_266.gz: (0x00000009) exists archived, archive_id:13
R1_267.gz: (0x00000009) exists archived, archive_id:13
R1_268.gz: (0x00000009) exists archived, archive_id:13
R1_269.gz: (0x00000009) exists archived, archive_id:13
R1_26.gz: (0x00000009) exists archived, archive_id:13
R1_270.gz: (0x00000009) exists archived, archive_id:13
R1_271.gz: (0x00000009) exists archived, archive_id:13
R1_272.gz: (0x00000009) exists archived, archive_id:13
R1_273.gz: (0x00000009) exists archived, archive_id:13
R1_274.gz: (0x00000009) exists archived, archive_id:13
R1_275.gz: (0x00000009) exists archived, archive_id:13
R1_276.gz: (0x00000009) exists archived, archive_id:13
R1_277.gz: (0x00000009) exists archived, archive_id:13
R1_278.gz: (0x00000009) exists archived, archive_id:13
R1_279.gz: (0x00000009) exists archived, archive_id:13
R1_27.gz: (0x00000009) exists archived, archive_id:13
R1_280.gz: (0x00000009) exists archived, archive_id:13
R1_281.gz: (0x00000009) exists archived, archive_id:13
R1_282.gz: (0x00000009) exists archived, archive_id:13
R1_283.gz: (0x00000009) exists archived, archive_id:13
R1_284.gz: (0x00000009) exists archived, archive_id:13
R1_285.gz: (0x00000009) exists archived, archive_id:13
R1_286.gz: (0x00000009) exists archived, archive_id:13
R1_287.gz: (0x00000009) exists archived, archive_id:13
R1_288.gz: (0x00000009) exists archived, archive_id:13
R1_289.gz: (0x00000009) exists archived, archive_id:13
R1_28.gz: (0x00000009) exists archived, archive_id:13
R1_290.gz: (0x00000009) exists archived, archive_id:13
R1_291.gz: (0x00000009) exists archived, archive_id:13
R1_292.gz: (0x00000009) exists archived, archive_id:13
R1_293.gz: (0x00000009) exists archived, archive_id:13
R1_294.gz: (0x00000009) exists archived, archive_id:13
R1_295.gz: (0x00000009) exists archived, archive_id:13
R1_296.gz: (0x00000009) exists archived, archive_id:13
R1_297.gz: (0x00000009) exists archived, archive_id:13
R1_298.gz: (0x00000009) exists archived, archive_id:13
R1_299.gz: (0x00000009) exists archived, archive_id:13
R1_29.gz: (0x00000009) exists archived, archive_id:13
R1_2.gz: (0x00000009) exists archived, archive_id:13
R1_300.gz: (0x00000009) exists archived, archive_id:13
R1_301.gz: (0x00000009) exists archived, archive_id:13
R1_302.gz: (0x00000009) exists archived, archive_id:13
R1_303.gz: (0x00000009) exists archived, archive_id:13
R1_304.gz: (0x00000009) exists archived, archive_id:13
R1_305.gz: (0x00000009) exists archived, archive_id:13
R1_306.gz: (0x00000009) exists archived, archive_id:13
R1_307.gz: (0x00000009) exists archived, archive_id:13
R1_308.gz: (0x00000009) exists archived, archive_id:13
R1_309.gz: (0x00000009) exists archived, archive_id:13
R1_30.gz: (0x00000009) exists archived, archive_id:13
R1_310.gz: (0x00000009) exists archived, archive_id:13
R1_311.gz: (0x00000009) exists archived, archive_id:13
R1_312.gz: (0x00000009) exists archived, archive_id:13
R1_313.gz: (0x00000009) exists archived, archive_id:13
R1_314.gz: (0x00000009) exists archived, archive_id:13
R1_315.gz: (0x00000009) exists archived, archive_id:13
R1_316.gz: (0x00000009) exists archived, archive_id:13
R1_317.gz: (0x00000009) exists archived, archive_id:13
R1_318.gz: (0x00000009) exists archived, archive_id:13
R1_319.gz: (0x00000009) exists archived, archive_id:13
R1_31.gz: (0x00000009) exists archived, archive_id:13
R1_320.gz: (0x00000009) exists archived, archive_id:13
R1_321.gz: (0x00000009) exists archived, archive_id:13
R1_322.gz: (0x00000009) exists archived, archive_id:13
R1_323.gz: (0x00000009) exists archived, archive_id:13
R1_324.gz: (0x00000009) exists archived, archive_id:13
R1_325.gz: (0x00000009) exists archived, archive_id:13
R1_326.gz: (0x00000009) exists archived, archive_id:13
R1_327.gz: (0x00000009) exists archived, archive_id:13
R1_328.gz: (0x00000009) exists archived, archive_id:13
R1_329.gz: (0x00000009) exists archived, archive_id:13
R1_32.gz: (0x00000009) exists archived, archive_id:13
R1_330.gz: (0x00000009) exists archived, archive_id:13
R1_331.gz: (0x00000009) exists archived, archive_id:13
R1_332.gz: (0x00000009) exists archived, archive_id:13
R1_333.gz: (0x00000009) exists archived, archive_id:13
R1_334.gz: (0x00000009) exists archived, archive_id:13
R1_335.gz: (0x00000009) exists archived, archive_id:13
R1_336.gz: (0x00000009) exists archived, archive_id:13
R1_337.gz: (0x00000009) exists archived, archive_id:13
R1_338.gz: (0x00000009) exists archived, archive_id:13
R1_339.gz: (0x00000009) exists archived, archive_id:13
R1_33.gz: (0x00000009) exists archived, archive_id:13
R1_340.gz: (0x00000009) exists archived, archive_id:13
R1_341.gz: (0x00000009) exists archived, archive_id:13
R1_342.gz: (0x00000009) exists archived, archive_id:13
R1_343.gz: (0x00000009) exists archived, archive_id:13
R1_344.gz: (0x00000009) exists archived, archive_id:13
R1_345.gz: (0x00000009) exists archived, archive_id:13
R1_346.gz: (0x00000009) exists archived, archive_id:13
R1_347.gz: (0x00000009) exists archived, archive_id:13
R1_348.gz: (0x00000009) exists archived, archive_id:13
R1_349.gz: (0x00000009) exists archived, archive_id:13
R1_34.gz: (0x00000009) exists archived, archive_id:13
R1_350.gz: (0x00000009) exists archived, archive_id:13
R1_351.gz: (0x00000009) exists archived, archive_id:13
R1_352.gz: (0x00000009) exists archived, archive_id:13
R1_353.gz: (0x00000009) exists archived, archive_id:13
R1_354.gz: (0x00000009) exists archived, archive_id:13
R1_355.gz: (0x00000009) exists archived, archive_id:13
R1_356.gz: (0x00000009) exists archived, archive_id:13
R1_357.gz: (0x00000009) exists archived, archive_id:13
R1_358.gz: (0x00000009) exists archived, archive_id:13
R1_359.gz: (0x00000009) exists archived, archive_id:13
R1_35.gz: (0x00000009) exists archived, archive_id:13
R1_360.gz: (0x00000009) exists archived, archive_id:13
R1_361.gz: (0x00000009) exists archived, archive_id:13
R1_362.gz: (0x00000009) exists archived, archive_id:13
R1_363.gz: (0x00000009) exists archived, archive_id:13
R1_364.gz: (0x00000009) exists archived, archive_id:13
R1_365.gz: (0x00000009) exists archived, archive_id:13
R1_366.gz: (0x00000009) exists archived, archive_id:13
R1_367.gz: (0x00000009) exists archived, archive_id:13
R1_368.gz: (0x00000009) exists archived, archive_id:13
R1_369.gz: (0x00000009) exists archived, archive_id:13
R1_36.gz: (0x00000009) exists archived, archive_id:13
... 2178 more lines ...
  480   Mon Jun 13 16:38:37 2022 NHMon 13 Jun
Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change
  479   Fri Jun 10 10:32:46 2022 NHFri 10 Jun
Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 
Attachment 1: scopePSU1.jpg
scopePSU1.jpg
Attachment 2: scopePSU2.jpg
scopePSU2.jpg
Attachment 3: scoepPSU3.jpg
scoepPSU3.jpg
Attachment 4: scope5V_DC.jpg
scope5V_DC.jpg
Attachment 5: scopeN6V_DC.jpg
scopeN6V_DC.jpg
Attachment 6: scope_7VDC.jpg
scope_7VDC.jpg
Attachment 7: scope_RFAnt.jpg
scope_RFAnt.jpg
Attachment 8: scopeRF_pickup.jpg
scopeRF_pickup.jpg
  478   Wed Jun 8 09:55:28 2022 NHWeds 8 May
10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  
Attachment 1: 2022-06-08_14-48-21_goodevents.png
2022-06-08_14-48-21_goodevents.png
Attachment 2: rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
  477   Wed Jun 1 08:00:18 2022 TD2xDSSSDs spectra layouts
Spectra layouts for 2x DSSSDs can be found in directory ~/LayOut/2xDSSSDs
  476   Wed Jun 1 07:35:13 2022 TDWednesday 1 June
Attachment 1 - DSSSD bias & leakage current - OK

Attachment 2 - ADC data item stats

Attachment 3 - FEE64 temps - OK

Attachments 4-7 - aida01, 02, 03 & 04 2.*.W spectra

Noise observed for asics #1 & #2 of aida02 consistent with no cable attached - confirmed by visual inspection - ribbon cable disconnected from Samtec header of adaptor PCB

To Do list

1) re-connect ribbon cable to aida02 adaptor PCB
2) good quality ground to snout - use screws at inter-stage
3) install heavy duty grounds
4) ... 


Attachment 8 - adc data item stats - slow comparator 0x64

Attachment 9 - per FEE64 rate spectra - slow comparator 0x64

Attachment 10 - per FEE64 rate spectra - slow comparator 0xa

-

13:30 - Attach heavy duty ground to AIDA 

Attachment 11 - per FEE64 rate spectra - slow comparator 0xa

Try power cycle as system ground changed significantly now

Try to reattach aida04 TTY connector - still wrong

Archive old tty logs (ttyUSB*) to zip and delete them, makes easier to track startup 

Attachment 12 - per FEE64 rate spectra - slow comparator 0xa

No major difference, aida01 quieter than before

aida05 has some missing channels - loose ERNI?

Attachements 13 & 14 - Waveforms

Temporarily raise to 0x64 and test merger for DTAS synchro test later... all works good to MBS 
Change NETVAR MERGE.LinksAvailable to 8

Notice one copper screw touching aluminium, add some kapton tape behind it to isolate it. No difference

Voltage diff between aida02,aida06 < 1 mV
Voltage diff between copper bars < 0.5 mV (but not 0)

A comment from DTAS people that they observed a large increase in noise recently - perhaps from accelerator
May indicate current power/gronud oddities?

Attachment 15 - Rates at 0x1f (= 310 keV)

DSSSD#1 reasonable (aida04 a bit bad), but mostly just pulser (25 Hz)

DSSSD#2 much worse esp. p+n sides (aida05, aida07)

ERNI connectors pushed in and feel fully in
Attachment 1: Screenshot_from_2022-06-01_08-33-30.png
Screenshot_from_2022-06-01_08-33-30.png
Attachment 2: Screenshot_from_2022-06-01_08-33-13.png
Screenshot_from_2022-06-01_08-33-13.png
Attachment 3: Screenshot_from_2022-06-01_08-32-44.png
Screenshot_from_2022-06-01_08-32-44.png
Attachment 4: Screenshot_from_2022-06-01_08-32-14.png
Screenshot_from_2022-06-01_08-32-14.png
Attachment 5: Screenshot_from_2022-06-01_08-31-04.png
Screenshot_from_2022-06-01_08-31-04.png
Attachment 6: Screenshot_from_2022-06-01_08-29-37.png
Screenshot_from_2022-06-01_08-29-37.png
Attachment 7: Screenshot_from_2022-06-01_08-27-41.png
Screenshot_from_2022-06-01_08-27-41.png
Attachment 8: Screenshot_from_2022-06-01_08-54-54.png
Screenshot_from_2022-06-01_08-54-54.png
Attachment 9: Screenshot_from_2022-06-01_08-54-34.png
Screenshot_from_2022-06-01_08-54-34.png
Attachment 10: Screenshot_from_2022-06-01_08-51-51.png
Screenshot_from_2022-06-01_08-51-51.png
Attachment 11: 2022-06-01_13-29-47_stats.png
2022-06-01_13-29-47_stats.png
Attachment 12: 2022-06-01_13-42-07_rates.png
2022-06-01_13-42-07_rates.png
Attachment 13: 2022-06-01_13-44-25_waves_pn.png
2022-06-01_13-44-25_waves_pn.png
Attachment 14: 2022-06-01_13-44-52_nn_waves.png
2022-06-01_13-44-52_nn_waves.png
Attachment 15: 2022-06-01_14-23-57_0x1f.png
2022-06-01_14-23-57_0x1f.png
  475   Tue May 31 17:09:35 2022 TDTuesday 31 May
2x MSL type BB18(DS)-1000 installed


Adaptor PCB LK config

all FEE64s LK2, LK4 fitted
n+n FEE64s LK1 fitted
bottom p+n FEE64s LK3 fitted


AIDA PSU config

top 1-3 2-4
bottom 5-7 6-8


DSSSD bias -> p+n FEE64 core, n+n FEE64 braid


Heavy duty grounds not connected to FEE64 adaptor PCBs


ASIC settings 2021Apr29-13-16-00
 slow comparator 0xa

BNC PB-5 settings
 amplitude 1.000V
 attenuation x1
 tau_d 1ms
 frequency 25Hz
 polarity +


Attachment 1 - DSSSD bias & Leakage current - OK

Attachment 2 - FEE64 temps OK

Attachment 3 - adc data item stats

Attachments 4-9 - system wide checks OK *except* aida07 fails ADC calibration

Attachments 10 & 11 - 1.8.W spectra 20us FSR

Attachments 12 & 13 - 1.8.L spectra - aida03 pulser peak width 155 ch FWHM

Attachment 14 - per FEE64 rate spectra



19.15 Attachments 15-18 - 2*W spectra for aida01, 02, 03 &04

Attachment 19 - per FEE64 rate spectra

Attachment 20 - adc data item stats
Attachment 1: Screenshot_from_2022-05-31_18-09-27.png
Screenshot_from_2022-05-31_18-09-27.png
Attachment 2: Screenshot_from_2022-05-31_18-09-49.png
Screenshot_from_2022-05-31_18-09-49.png
Attachment 3: Screenshot_from_2022-05-31_18-10-18.png
Screenshot_from_2022-05-31_18-10-18.png
Attachment 4: Screenshot_from_2022-05-31_18-17-27.png
Screenshot_from_2022-05-31_18-17-27.png
Attachment 5: Screenshot_from_2022-05-31_18-17-02.png
Screenshot_from_2022-05-31_18-17-02.png
Attachment 6: Screenshot_from_2022-05-31_18-16-43.png
Screenshot_from_2022-05-31_18-16-43.png
Attachment 7: Screenshot_from_2022-05-31_18-16-28.png
Screenshot_from_2022-05-31_18-16-28.png
Attachment 8: Screenshot_from_2022-05-31_18-16-12.png
Screenshot_from_2022-05-31_18-16-12.png
Attachment 9: Screenshot_from_2022-05-31_18-15-13.png
Screenshot_from_2022-05-31_18-15-13.png
Attachment 10: Screenshot_from_2022-05-31_18-32-34.png
Screenshot_from_2022-05-31_18-32-34.png
Attachment 11: Screenshot_from_2022-05-31_18-31-56.png
Screenshot_from_2022-05-31_18-31-56.png
Attachment 12: Screenshot_from_2022-05-31_18-30-55.png
Screenshot_from_2022-05-31_18-30-55.png
Attachment 13: Screenshot_from_2022-05-31_18-29-53.png
Screenshot_from_2022-05-31_18-29-53.png
Attachment 14: Screenshot_from_2022-05-31_18-28-56.png
Screenshot_from_2022-05-31_18-28-56.png
Attachment 15: Screenshot_from_2022-05-31_19-13-51.png
Screenshot_from_2022-05-31_19-13-51.png
Attachment 16: Screenshot_from_2022-05-31_19-13-03.png
Screenshot_from_2022-05-31_19-13-03.png
Attachment 17: Screenshot_from_2022-05-31_19-12-19.png
Screenshot_from_2022-05-31_19-12-19.png
Attachment 18: Screenshot_from_2022-05-31_19-11-24.png
Screenshot_from_2022-05-31_19-11-24.png
Attachment 19: Screenshot_from_2022-05-31_19-09-10.png
Screenshot_from_2022-05-31_19-09-10.png
Attachment 20: Screenshot_from_2022-05-31_19-06-59.png
Screenshot_from_2022-05-31_19-06-59.png
  474   Tue May 31 09:05:00 2022 NH, TDTuesday 31 May
DSSD stack will be as follows

1      3208-7  1019       125              upstream
2      3208-13 1020       120              downstream

First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance

3208-7: 
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

3208-13:
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

Conclusion: All bias bond wires are intact on both DSSDs

Visual inspection is both DSSDs look good with no damage or bond wire issues

Installed in snout: full stack

3 mm bPlas 1
17 mm Air
1 mm AIDA 1  (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2
  473   Wed May 25 14:28:26 2022 NHWed 25 May
The n+n FEEs were all removed to adjust the platform back to the single configuration

After some confusion we found the orientation of the frame to position the single detectors and moved back

Faulty AIDA cards aida05 and aida14 were removed to be returned backi (ASIC damage)

The cables for aida10 and aida12 were moved back to the left-right side and the n+n cards were installed and reocnnected

p+n cards moved to bring aida11 and aida09 above aida03/aida07 and aida01/aida05. Also move a different card to replace old aida05

While doing this it was noticed the HDMI for aida13 was very loose and actually fell off (I guess the cable was pulled incorrectly)
It has been removed and will eb sent back to be resoldered 

The copper grounding bars and ITEM frame were removed due to conflicting with the single snout layout
The bolts are slightly too short to hold the frame where it won't conflict so we loko for longer bolts to reattach the system
Some new ground cables will be made to reach the extra n+n FEEs

Aida13-Aida16 removed from the PSU 

fig1: layout of snout frame in "single" mode (alu block configuration)


MAC Addresses

aida01 - ba-8a
aida05 - 0d:15
aida09 - ee-01

aida04 - f6-b7
aida08 - ba-2b
aida12 - ba:89 

aida03 - d8-21
aida07 - 41:b4:0c
aida11 - f6-5a

aida02 - ba:22
aida06 - ee:72
aida10 - cf-ac [mistakenly swapped with aida12]

Second switch will stay in S4 and has now been screwed in properly below the switch 1 and the hdmi cables entrance

DHCP *not* updated

Relay is powered *off* so FEEs cannot be turned on, alL HV channels disabled

Tomorrow is a holiday so no work will be happening @ GSI
Attachment 1: aida_single_alu.jpg
aida_single_alu.jpg
  472   Tue May 24 08:35:00 2022 NHTue 24 May
09:00 - Remove bias of DSSDs  and pwoer of FEEs

Remove snout from S4 so DEGAS frame can be mechnically removed

Snout now in FRS Messhuette 
  471   Mon May 23 16:41:45 2022 TDMonday 23 May
17.40 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-23_17-39-23.png
Screenshot_from_2022-05-23_17-39-23.png
Attachment 2: Screenshot_from_2022-05-23_17-39-58.png
Screenshot_from_2022-05-23_17-39-58.png
Attachment 3: Screenshot_from_2022-05-23_17-40-33.png
Screenshot_from_2022-05-23_17-40-33.png
  470   Sun May 22 14:12:55 2022 TDSunday 22 May
15.11 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-22_15-10-25.png
Screenshot_from_2022-05-22_15-10-25.png
Attachment 2: Screenshot_from_2022-05-22_15-10-48.png
Screenshot_from_2022-05-22_15-10-48.png
Attachment 3: Screenshot_from_2022-05-22_15-11-35.png
Screenshot_from_2022-05-22_15-11-35.png
  469   Sat May 21 15:30:45 2022 TDSaturday 21 May
16.31 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-21_16-27-58.png
Screenshot_from_2022-05-21_16-27-58.png
Attachment 2: Screenshot_from_2022-05-21_16-28-59.png
Screenshot_from_2022-05-21_16-28-59.png
Attachment 3: Screenshot_from_2022-05-21_16-29-36.png
Screenshot_from_2022-05-21_16-29-36.png
  468   Fri May 20 16:22:01 2022 TDFriday 20 May
17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-20_17-19-21.png
Screenshot_from_2022-05-20_17-19-21.png
Attachment 2: Screenshot_from_2022-05-20_17-19-35.png
Screenshot_from_2022-05-20_17-19-35.png
Attachment 3: Screenshot_from_2022-05-20_17-20-22.png
Screenshot_from_2022-05-20_17-20-22.png
  467   Thu May 19 17:00:46 2022 TDThursday 19 May
18.01 DAQ stopped

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachment 2 - DSSSD bias & leakage current

Attachment 3 - FEE64 temps - OK

Attachment 4 - ADC data item stats
Attachment 1: Screenshot_from_2022-05-19_17-59-41.png
Screenshot_from_2022-05-19_17-59-41.png
Attachment 2: Screenshot_from_2022-05-19_17-58-21.png
Screenshot_from_2022-05-19_17-58-21.png
Attachment 3: Screenshot_from_2022-05-19_17-58-52.png
Screenshot_from_2022-05-19_17-58-52.png
Attachment 4: Screenshot_from_2022-05-19_17-59-20.png
Screenshot_from_2022-05-19_17-59-20.png
  466   Wed May 18 15:29:50 2022 TDWednesday 18 May
      file no storage mode, no MBS data transfer

      ADC control register 0x0 

      std disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xa

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - ADC data item stats

Attachment 4 - FEE64 temps - OK

16.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      baseline counters
      system wide checks - all OK *except* aida07 WR error 0x10

Attachment 5 - per FEE64 rate spectra

Attachmnents 6 & 7 - per FEE64 1.8.W spectra 20us FSR
Attachment 1: Screenshot_from_2022-05-18_16-24-18.png
Screenshot_from_2022-05-18_16-24-18.png
Attachment 2: Screenshot_from_2022-05-18_16-24-41.png
Screenshot_from_2022-05-18_16-24-41.png
Attachment 3: Screenshot_from_2022-05-18_16-26-13.png
Screenshot_from_2022-05-18_16-26-13.png
Attachment 4: Screenshot_from_2022-05-18_16-26-53.png
Screenshot_from_2022-05-18_16-26-53.png
Attachment 5: Screenshot_from_2022-05-18_16-36-47.png
Screenshot_from_2022-05-18_16-36-47.png
Attachment 6: Screenshot_from_2022-05-18_16-39-48.png
Screenshot_from_2022-05-18_16-39-48.png
Attachment 7: Screenshot_from_2022-05-18_16-52-18.png
Screenshot_from_2022-05-18_16-52-18.png
Attachment 8: Screenshot_from_2022-05-18_16-53-32.png
Screenshot_from_2022-05-18_16-53-32.png
  465   Tue May 17 16:53:20 2022 TDTuesday 17 May 16.00-00.00
17.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      file no storage mode, no MBS data transfer

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0x80

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2-10 - NewMerger stats

Attachments 11-13 - iptraf, TapeServer, NewMerger

Attachment 14 - per Fee64 rate spectra

Attachment 15 - lost activity  monitor

Attachment 16 - ASIC settings aida01

Attachments 17-22 - adc, pause, resume & correlation scaler data items, push, flush, aida01 & aida06 stats

Attachment 23 - FEE64 temperatures OK

Attachment 24 - DSSSD bias & leakage current
Attachment 1: Screenshot_from_2022-05-17_17-52-15.png
Screenshot_from_2022-05-17_17-52-15.png
Attachment 2: Screenshot_from_2022-05-17_17-50-50.png
Screenshot_from_2022-05-17_17-50-50.png
Attachment 3: Screenshot_from_2022-05-17_17-49-18.png
Screenshot_from_2022-05-17_17-49-18.png
Attachment 4: Screenshot_from_2022-05-17_17-48-24.png
Screenshot_from_2022-05-17_17-48-24.png
Attachment 5: Screenshot_from_2022-05-17_17-47-35.png
Screenshot_from_2022-05-17_17-47-35.png
Attachment 6: Screenshot_from_2022-05-17_17-47-12.png
Screenshot_from_2022-05-17_17-47-12.png
Attachment 7: Screenshot_from_2022-05-17_17-46-19.png
Screenshot_from_2022-05-17_17-46-19.png
Attachment 8: Screenshot_from_2022-05-17_17-45-34.png
Screenshot_from_2022-05-17_17-45-34.png
Attachment 9: Screenshot_from_2022-05-17_17-45-10.png
Screenshot_from_2022-05-17_17-45-10.png
Attachment 10: Screenshot_from_2022-05-17_17-44-34.png
Screenshot_from_2022-05-17_17-44-34.png
Attachment 11: Screenshot_from_2022-05-17_17-44-03.png
Screenshot_from_2022-05-17_17-44-03.png
Attachment 12: Screenshot_from_2022-05-17_17-43-37.png
Screenshot_from_2022-05-17_17-43-37.png
Attachment 13: Screenshot_from_2022-05-17_17-42-58.png
Screenshot_from_2022-05-17_17-42-58.png
Attachment 14: Screenshot_from_2022-05-17_17-42-40.png
Screenshot_from_2022-05-17_17-42-40.png
Attachment 15: Screenshot_from_2022-05-17_17-40-53.png
Screenshot_from_2022-05-17_17-40-53.png
Attachment 16: Screenshot_from_2022-05-17_17-40-08.png
Screenshot_from_2022-05-17_17-40-08.png
Attachment 17: Screenshot_from_2022-05-17_17-39-20.png
Screenshot_from_2022-05-17_17-39-20.png
Attachment 18: Screenshot_from_2022-05-17_17-38-52.png
Screenshot_from_2022-05-17_17-38-52.png
Attachment 19: Screenshot_from_2022-05-17_17-38-20.png
Screenshot_from_2022-05-17_17-38-20.png
Attachment 20: Screenshot_from_2022-05-17_17-37-47.png
Screenshot_from_2022-05-17_17-37-47.png
Attachment 21: Screenshot_from_2022-05-17_17-37-19.png
Screenshot_from_2022-05-17_17-37-19.png
Attachment 22: Screenshot_from_2022-05-17_17-36-50.png
Screenshot_from_2022-05-17_17-36-50.png
Attachment 23: Screenshot_from_2022-05-17_17-36-14.png
Screenshot_from_2022-05-17_17-36-14.png
Attachment 24: Screenshot_from_2022-05-17_17-35-42.png
Screenshot_from_2022-05-17_17-35-42.png
  464   Tue May 17 06:58:51 2022 OHTuesday 17th May
08:00 Still have beam for the time being.
      Currently on file R4_2049

      FEE Stats - attachment 1
      FEE Temps ok - attachment 2
      Bias and leakage currents - attachment 3
      System wide checks all ok *except aida09 fails clock check with value 6 which is ok*

      Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement.
      Once they have left the area and are taking data a pulser walkthrough will be performed.
      Patrick can then have access to the system

08:16 Beam taken away
      Data taken stopped on file R4_2055
      Will perform a pulser walkthrough now

      Pulser walkthrough started on R5
      1V->0.1V in 0.1V steps
      Pulser setting - attachment 4
      p+n pulser peaks - attachment 5
      n+n pulser "peaks" - attachment 6
Attachment 1: Screenshot_from_2022-05-17_08-00-50.png
Screenshot_from_2022-05-17_08-00-50.png
Attachment 2: Screenshot_from_2022-05-17_08-01-13.png
Screenshot_from_2022-05-17_08-01-13.png
Attachment 3: Screenshot_from_2022-05-17_08-01-25.png
Screenshot_from_2022-05-17_08-01-25.png
Attachment 4: Screenshot_from_2022-05-17_08-49-02.png
Screenshot_from_2022-05-17_08-49-02.png
Attachment 5: Screenshot_from_2022-05-17_08-51-19.png
Screenshot_from_2022-05-17_08-51-19.png
Attachment 6: Screenshot_from_2022-05-17_08-51-52.png
Screenshot_from_2022-05-17_08-51-52.png
  463   Mon May 16 23:02:54 2022 AMTuesday 17th May 00:00-08:00

01:00:     Attachments 1-3, System checks good

03:00:     Attachments 4-6, System checks good

05:00:     Attachments 7-9, System checks good

07:00:     Attachments 10-12, System checks good

Attachment 1: 220512_0604_biasLeakage.png
220512_0604_biasLeakage.png
Attachment 2: 220512_0603_temps.png
220512_0603_temps.png
Attachment 3: 220512_0601_stats.png
220512_0601_stats.png
Attachment 4: 220517_0300_temps.png
220517_0300_temps.png
Attachment 5: 220517_0300_stats.png
220517_0300_stats.png
Attachment 6: 220517_0300_biasLeakage.png
220517_0300_biasLeakage.png
Attachment 7: 220517_0500_temps.png
220517_0500_temps.png
Attachment 8: 220517_0500_stats.png
220517_0500_stats.png
Attachment 9: 220517_0500_biasLeakage.png
220517_0500_biasLeakage.png
Attachment 10: 220517_0700_temps.png
220517_0700_temps.png
Attachment 11: 220517_0700_stats.png
220517_0700_stats.png
Attachment 12: 220517_0700_biasLeakage.png
220517_0700_biasLeakage.png
  462   Mon May 16 17:23:27 2022 TDNewMerger stats
Attachment 1: Screenshot_from_2022-05-16_18-17-10.png
Screenshot_from_2022-05-16_18-17-10.png
Attachment 2: Screenshot_from_2022-05-16_18-17-50.png
Screenshot_from_2022-05-16_18-17-50.png
Attachment 3: Screenshot_from_2022-05-16_18-18-24.png
Screenshot_from_2022-05-16_18-18-24.png
Attachment 4: Screenshot_from_2022-05-16_18-20-02.png
Screenshot_from_2022-05-16_18-20-02.png
  461   Mon May 16 15:12:50 2022 RDP, MAMonday 16th May 16:00-00:00

16:00 Took over the shift from Magda.

17:30 Updates: Attached (1-3)

19:30 Updates: Attached (4-6)

21:30 Updates: Attached (7-9)

23:00 Updates: Attached (10-12)

 

 

 

Attachment 1: Statistics_2022-05-16_17-32-40.png
Statistics_2022-05-16_17-32-40.png
Attachment 2: Temperature_2022-05-16_17-34-15.png
Temperature_2022-05-16_17-34-15.png
Attachment 3: LeakageCurrent_2022-05-16_17-35-05.png
LeakageCurrent_2022-05-16_17-35-05.png
Attachment 4: Statistics2022-05-16_19-31-28.png
Statistics2022-05-16_19-31-28.png
Attachment 5: Temperature2022-05-16_19-32-39.png
Temperature2022-05-16_19-32-39.png
Attachment 6: LeakageCurrent_2022-05-16_19-30-41.png
LeakageCurrent_2022-05-16_19-30-41.png
Attachment 7: Statistics2022-05-16_21-27-17.png
Statistics2022-05-16_21-27-17.png
Attachment 8: Temperature_2022-05-16_21-28-21.png
Temperature_2022-05-16_21-28-21.png
Attachment 9: LeakageCurrent2022-05-16_21-26-15.png
LeakageCurrent2022-05-16_21-26-15.png
Attachment 10: Statistics2022-05-16_22-59-14.png
Statistics2022-05-16_22-59-14.png
Attachment 11: Temperature_2022-05-16_23-00-18.png
Temperature_2022-05-16_23-00-18.png
Attachment 12: LeakageCurrent_2022-05-16_22-58-06.png
LeakageCurrent_2022-05-16_22-58-06.png
  460   Mon May 16 14:07:06 2022 TDLost activity monitor
Attachment 1: Screenshot_from_2022-05-16_15-04-56.png
Screenshot_from_2022-05-16_15-04-56.png
  459   Mon May 16 07:03:16 2022 MS, OHMonday 16th May 08:00-16:00

08:00 Took over the night shift from Tom (Attachments 1-3)

10:00: Attachement 4-6

around 10:30: Beam went off (Attachment 7)

11:00: Beam is back

12:00 Attachments 8-10

14:00 Attachements 11-13

15:50 Attachments 14-17

Attachment 1: Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-16_08-01-20.png
Screenshot_from_2022-05-16_08-01-20.png
Attachment 4: Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-05-16_10-01-47.png
Screenshot_from_2022-05-16_10-01-47.png
Attachment 7: Screenshot_2022-05-16_at_10-47-01_ucesb.png
Screenshot_2022-05-16_at_10-47-01_ucesb.png
Attachment 8: Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Attachment 9: Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 10: Screenshot_from_2022-05-16_12-00-57.png
Screenshot_from_2022-05-16_12-00-57.png
Attachment 11: Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-05-16_14-01-56.png
Screenshot_from_2022-05-16_14-01-56.png
Attachment 14: Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Attachment 17: Screenshot_from_2022-05-16_15-49-34.png
Screenshot_from_2022-05-16_15-49-34.png
  458   Sun May 15 23:00:08 2022 & TDMonday 16 May 00:00-08:00
00.01 DAQ continues
      file S450/R4_1319

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Disk space OK - /media/SecondDrive

[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  389M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  4.1T  2.8T  61% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

00.03 all histograms zero'd
      system wide checks counter baseline


00.08 check ASIC control - all FEE64s all ASICs

Attachments 1 & 2 - DSSSD bias & Leakage current - OK
                    grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps OK

Attachments 4-9 - adc, pause, resume & correlation scaler data items, push, flush stats

Attachments 10-16 - TapeServer, NewMerger, NewMerger stats

Attachment 17 - ucesb

04.08

Attachment 18 - DSSSD bias & Leakage current - OK

Attachment 19 - FEE64 temps OK

Attachment 20 - adc data item stats

Attachments 21 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

06.36

Attachment 22 - DSSSD bias & Leakage current - OK

Attachment 23 - FEE64 temps OK

Attachment 24 - adc data item stats

Attachments 25 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6
Attachment 1: Screenshot_from_2022-05-16_00-10-13.png
Screenshot_from_2022-05-16_00-10-13.png
Attachment 2: Screenshot_from_2022-05-16_00-10-43.png
Screenshot_from_2022-05-16_00-10-43.png
Attachment 3: Screenshot_from_2022-05-16_00-11-18.png
Screenshot_from_2022-05-16_00-11-18.png
Attachment 4: Screenshot_from_2022-05-16_00-11-50.png
Screenshot_from_2022-05-16_00-11-50.png
Attachment 5: Screenshot_from_2022-05-16_00-12-27.png
Screenshot_from_2022-05-16_00-12-27.png
Attachment 6: Screenshot_from_2022-05-16_00-12-54.png
Screenshot_from_2022-05-16_00-12-54.png
Attachment 7: Screenshot_from_2022-05-16_00-13-25.png
Screenshot_from_2022-05-16_00-13-25.png
Attachment 8: Screenshot_from_2022-05-16_00-13-55.png
Screenshot_from_2022-05-16_00-13-55.png
Attachment 9: Screenshot_from_2022-05-16_00-14-22.png
Screenshot_from_2022-05-16_00-14-22.png
Attachment 10: Screenshot_from_2022-05-16_00-15-46.png
Screenshot_from_2022-05-16_00-15-46.png
Attachment 11: Screenshot_from_2022-05-16_00-16-08.png
Screenshot_from_2022-05-16_00-16-08.png
Attachment 12: Screenshot_from_2022-05-16_00-16-36.png
Screenshot_from_2022-05-16_00-16-36.png
Attachment 13: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 14: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 15: Screenshot_from_2022-05-16_00-17-32.png
Screenshot_from_2022-05-16_00-17-32.png
Attachment 16: Screenshot_from_2022-05-16_00-17-55.png
Screenshot_from_2022-05-16_00-17-55.png
Attachment 17: Screenshot_from_2022-05-16_00-18-45.png
Screenshot_from_2022-05-16_00-18-45.png
Attachment 18: Screenshot_from_2022-05-16_04-08-34.png
Screenshot_from_2022-05-16_04-08-34.png
Attachment 19: Screenshot_from_2022-05-16_04-09-07.png
Screenshot_from_2022-05-16_04-09-07.png
Attachment 20: Screenshot_from_2022-05-16_04-09-57.png
Screenshot_from_2022-05-16_04-09-57.png
Attachment 21: Screenshot_from_2022-05-16_04-10-27.png
Screenshot_from_2022-05-16_04-10-27.png
Attachment 22: Screenshot_from_2022-05-16_06-35-32.png
Screenshot_from_2022-05-16_06-35-32.png
Attachment 23: Screenshot_from_2022-05-16_06-36-37.png
Screenshot_from_2022-05-16_06-36-37.png
Attachment 24: Screenshot_from_2022-05-16_06-37-04.png
Screenshot_from_2022-05-16_06-37-04.png
Attachment 25: Screenshot_from_2022-05-16_06-37-58.png
Screenshot_from_2022-05-16_06-37-58.png
  457   Sun May 15 15:51:49 2022 MLSunday 15th May 16:00-0:00

Status at 16:45 (CET)

The expreiment continues to run smoothly.

AIDA Stats look ok - Attachment 1

AIDA Temperature ok - Attachment 2

AIDA Leakage current: ok - Attachment 3

System wide check:

Clock: 13 passed, 1 failed (aida09)

ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)

White rabbit decoder: 14 passed, 0 failed.

FPGA timestamp: 14 passed, 0 failed.

 

Status at 18:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 4

AIDA Temperature ok - Attachment 5

AIDA Leakage current: increase slightly - Attachment 6

System wide check: No changes, same as above

 

Status at 20:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 7

AIDA Temperature ok - Attachment 8

AIDA Leakage current: increase slightly - Attachment 9

System wide check: No changes, same as above

 

Status at 22:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 10

AIDA Temperature ok - Attachment 11

AIDA Leakage current: increase slightly - Attachment 12

System wide check: No changes, same as above

Attachment 1: Stats_Screenshot_from_2022-05-15_16-49-06.png
Stats_Screenshot_from_2022-05-15_16-49-06.png
Attachment 2: Temp_Screenshot_from_2022-05-15_16-48-31.png
Temp_Screenshot_from_2022-05-15_16-48-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-15_16-47-44.png
HV_LC_Screenshot_from_2022-05-15_16-47-44.png
Attachment 4: Stats_Screenshot_from_2022-05-15_18-43-16.png
Stats_Screenshot_from_2022-05-15_18-43-16.png
Attachment 5: Temp_Screenshot_from_2022-05-15_18-44-06.png
Temp_Screenshot_from_2022-05-15_18-44-06.png
Attachment 6: HV_LC_Screenshot_from_2022-05-15_18-42-23.png
HV_LC_Screenshot_from_2022-05-15_18-42-23.png
Attachment 7: Stats_Screenshot_from_2022-05-15_20-42-03.png
Stats_Screenshot_from_2022-05-15_20-42-03.png
Attachment 8: Temp_Screenshot_from_2022-05-15_20-42-42.png
Temp_Screenshot_from_2022-05-15_20-42-42.png
Attachment 9: HV_LC_Screenshot_from_2022-05-15_20-36-28.png
HV_LC_Screenshot_from_2022-05-15_20-36-28.png
Attachment 10: Stats_Screenshot_from_2022-05-15_22-43-37.png
Stats_Screenshot_from_2022-05-15_22-43-37.png
Attachment 11: Temp_Screenshot_from_2022-05-15_22-44-20.png
Temp_Screenshot_from_2022-05-15_22-44-20.png
Attachment 12: HV_LC_Screenshot_from_2022-05-15_22-42-36.png
HV_LC_Screenshot_from_2022-05-15_22-42-36.png
  456   Sun May 15 07:40:52 2022 OHSunday 15th May 08:00-16:00
08:40 FEE Temperatures ok - attachment 1
      FEE statistics - Attachment 2
      Bias and leakage currents ok - attachment 3
      ASIC check ok
      All system wide checks ok - Except aida09 fails clock check with bit 6

11:00 FEE statistics - attachment 4
      FEE temperatures all ok - attachment 5
      BIAS and leakage currents - attachment 6
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1030
      13.1 MB/s to disk
      4.8 million items per second
      
13:13 FEE statistics - attachment 7
      FEE Temperatures all ok - attahcment 8
      Bias and leakage currents ok - attachment 9
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1078
      13.1 MB/s to disk
      4.8 million items per second

15:35 Analysis of R4_1128 - attachment 10
Attachment 1: Screenshot_from_2022-05-15_08-38-04.png
Screenshot_from_2022-05-15_08-38-04.png
Attachment 2: Screenshot_from_2022-05-15_08-38-34.png
Screenshot_from_2022-05-15_08-38-34.png
Attachment 3: Screenshot_from_2022-05-15_08-39-12.png
Screenshot_from_2022-05-15_08-39-12.png
Attachment 4: Screenshot_from_2022-05-15_11-00-32.png
Screenshot_from_2022-05-15_11-00-32.png
Attachment 5: Screenshot_from_2022-05-15_11-00-50.png
Screenshot_from_2022-05-15_11-00-50.png
Attachment 6: Screenshot_from_2022-05-15_11-01-02.png
Screenshot_from_2022-05-15_11-01-02.png
Attachment 7: Screenshot_from_2022-05-15_13-13-50.png
Screenshot_from_2022-05-15_13-13-50.png
Attachment 8: Screenshot_from_2022-05-15_13-14-12.png
Screenshot_from_2022-05-15_13-14-12.png
Attachment 9: Screenshot_from_2022-05-15_13-14-31.png
Screenshot_from_2022-05-15_13-14-31.png
Attachment 10: R4_1128
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259195371 ( 1637875.8 Hz)
        Other data format:    2724629 (   17217.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2455 (      15.5 Hz)
                               RESUME:       2456 (      15.5 Hz)
                              SYNC100:      32590 (     205.9 Hz)
                              WR48-63:      32590 (     205.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2654538 (   16774.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      38026 (     240.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      158.251 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                4.735                0.000
  2                0.000                0.000
  3               16.674                0.000
  4                0.000                0.000
  5               77.034                0.000
  6                0.000                0.000
  7               33.677                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000               12.872
 11               17.247                0.000
 12                0.000                0.000
 13                0.000                8.200
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7464928       2802          0          0          0          0        927        927          0        948          0        661
  1   38695355      10162          0          0        254        254       4827       4827          0          0          0      32843
  2    6890959    1397581          0          0          0          0       1028       1028          0    1395525          0        633
  3   41303989    1261446          0          0        398        398       5227       5227          0    1250196          0        331
  4    4066993       1070          0          0          0          0        535        535          0          0          0        242
  5   37291979      10883          0          0        737        738       4704       4704          0          0          0        222
  6   12390385       7719          0          0          0          0       1626       1626          0       4467          0        394
  7   42901955      15088          0          0        626        626       5217       5217          0       3402          0        167
  8   10872099       2752          0          0          0          0       1376       1376          0          0          0        592
  9    4061050       1014          0          0          0          0        507        507          0          0          0        432
 10    2113468        548          0          0          0          0        274        274          0          0          0        291
 11   41561869      11172          0          0        440        440       5146       5146          0          0          0        468
 12    6798748       1722          0          0          0          0        861        861          0          0          0        436
 13    2781594        670          0          0          0          0        335        335          0          0          0        314
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.348s (  959.588 blocks/s,  59.974 Mb/s)
  455   Sat May 14 23:02:33 2022 TDSunday 15 May 00:00-08:00
Pb beam c. 1e+9/spill

00.02 zero all histograms
      system wide checks - baseline counters

00.08 ASIC check control - all FEE64s, all ASICs

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2 & 3 - ucesb - AIDA DSSSD #1 c. 10-20Hz peak rate on spill, c. 0Hz off spill

Attachment 4 - DSSSD bias & Leakage current - OK

Attachment 5 - FEE64 temps OK

Attachment 6-13 - adc, pause, resume & correlation scaler data items, push, flush, aida01, aida06

Attachments 14-18 - system wide checks - all OK *except* aida09 clock fail status 6

Attachments 19-25 - iptraf, TapeServer, NewMerger, NewMerger stats



00.44 file S450/R4_792

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

00.51 disk space OK - c. 3' per data file

[npg@aidas-gsi S450]$ ls -l /
total 40
lrwxrwxrwx.   1 root root     7 Oct 18  2021 bin -> usr/bin
dr-xr-xr-x.   5 root root  4096 May 12 18:31 boot
drwxr-xr-x.  23 root root  4100 May 12 18:31 dev
drwxr-xr-x. 145 root root  8192 May 12 18:31 etc
drwxr-xr-x.   4 root root    56 Jan 28 14:56 home
lrwxrwxrwx.   1 root root     7 Oct 18  2021 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Oct 18  2021 lib64 -> usr/lib64
drwxr-xr-x.   5 root root   113 Apr 28 11:58 media
lrwxrwxrwx.   1 root root    45 Oct 18  2021 MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119
drwxr-xr-x.   2 root root     6 Apr 11  2018 mnt
drwxr-xr-x.   3 root root    16 Oct 18  2021 opt
dr-xr-xr-x. 464 root root     0 May 12 18:30 proc
dr-xr-x---.  10 root root  4096 May 13 11:47 root
drwxr-xr-x.  44 root root  1360 May 14 06:34 run
lrwxrwxrwx.   1 root root     8 Oct 18  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root root     6 Apr 11  2018 srv
dr-xr-xr-x.  13 root root     0 May 12 18:31 sys
lrwxrwxrwx.   1 root root    28 Apr 28 14:19 TapeData -> /media/SecondDrive/TapeData/
drwxrwxrwt.  40 root root 12288 May 15 00:49 tmp
drwxr-xr-x.  13 root root   155 Oct 18  2021 usr
drwxr-xr-x.  21 root root  4096 Oct 18  2021 var
[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  373M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  3.1T  3.8T  46% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

Attachment 26 - analysis data file R4_792

Attachments 27-28 - per FEE64 rate & stat spectra - shows distro HEC events

Attachments 29-30 - per FEE64 1.8.L spectra 
 pulser peak widths aida01 134 ch FWHM, aida04 393 ch FWHM

Attachments 31-32 - per FEE64 1.8.H spectra

Attachments 33-34 - aida02 & aida04 1.*.H spectra

04.13

Attachment 35 - DSSSD bias & Leakage current - OK

Attachment 36 - FEE64 temps OK

Attachment 37 - adc data item stats

Attachments 38 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6


07.05

Attachment 39 - DSSSD bias & Leakage current - OK

Attachment 40 - FEE64 temps OK

Attachment 41 - adc data item stats

Attachments 42 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

Attachments 43-48 - aida02 & aida04 & aida06 & aida08 & 1.*.H spectra
Attachment 1: Screenshot_from_2022-05-15_00-03-33.png
Screenshot_from_2022-05-15_00-03-33.png
Attachment 2: Screenshot_from_2022-05-15_00-03-58.png
Screenshot_from_2022-05-15_00-03-58.png
Attachment 3: Screenshot_from_2022-05-15_00-04-18.png
Screenshot_from_2022-05-15_00-04-18.png
Attachment 4: Screenshot_from_2022-05-15_00-04-39.png
Screenshot_from_2022-05-15_00-04-39.png
Attachment 5: Screenshot_from_2022-05-15_00-05-10.png
Screenshot_from_2022-05-15_00-05-10.png
Attachment 6: Screenshot_from_2022-05-15_00-07-25.png
Screenshot_from_2022-05-15_00-07-25.png
Attachment 7: Screenshot_from_2022-05-15_00-16-59.png
Screenshot_from_2022-05-15_00-16-59.png
Attachment 8: Screenshot_from_2022-05-15_00-17-52.png
Screenshot_from_2022-05-15_00-17-52.png
Attachment 9: Screenshot_from_2022-05-15_00-18-55.png
Screenshot_from_2022-05-15_00-18-55.png
Attachment 10: Screenshot_from_2022-05-15_00-19-35.png
Screenshot_from_2022-05-15_00-19-35.png
Attachment 11: Screenshot_from_2022-05-15_00-20-14.png
Screenshot_from_2022-05-15_00-20-14.png
Attachment 12: Screenshot_from_2022-05-15_00-20-57.png
Screenshot_from_2022-05-15_00-20-57.png
Attachment 13: Screenshot_from_2022-05-15_00-21-32.png
Screenshot_from_2022-05-15_00-21-32.png
Attachment 14: Screenshot_from_2022-05-15_00-26-13.png
Screenshot_from_2022-05-15_00-26-13.png
Attachment 15: Screenshot_from_2022-05-15_00-26-49.png
Screenshot_from_2022-05-15_00-26-49.png
Attachment 16: Screenshot_from_2022-05-15_00-27-19.png
Screenshot_from_2022-05-15_00-27-19.png
Attachment 17: Screenshot_from_2022-05-15_00-28-00.png
Screenshot_from_2022-05-15_00-28-00.png
Attachment 18: Screenshot_from_2022-05-15_00-28-45.png
Screenshot_from_2022-05-15_00-28-45.png
Attachment 19: Screenshot_from_2022-05-15_00-33-46.png
Screenshot_from_2022-05-15_00-33-46.png
Attachment 20: Screenshot_from_2022-05-15_00-35-33.png
Screenshot_from_2022-05-15_00-35-33.png
Attachment 21: Screenshot_from_2022-05-15_00-36-00.png
Screenshot_from_2022-05-15_00-36-00.png
Attachment 22: Screenshot_from_2022-05-15_00-36-26.png
Screenshot_from_2022-05-15_00-36-26.png
Attachment 23: Screenshot_from_2022-05-15_00-36-58.png
Screenshot_from_2022-05-15_00-36-58.png
Attachment 24: Screenshot_from_2022-05-15_00-37-25.png
Screenshot_from_2022-05-15_00-37-25.png
Attachment 25: Screenshot_from_2022-05-15_00-37-46.png
Screenshot_from_2022-05-15_00-37-46.png
Attachment 26: R4_792
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250102849 ( 1597022.2 Hz)
        Other data format:   11817151 (   75458.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2692 (      17.2 Hz)
                               RESUME:       2692 (      17.2 Hz)
                              SYNC100:      32584 (     208.1 Hz)
                              WR48-63:      32584 (     208.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11746599 (   75007.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      48170 (     307.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      156.606 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                6.802                0.000
  2                0.000                0.000
  3               28.710                0.000
  4                0.000                0.000
  5               76.548                0.000
  6                0.000                0.000
  7               36.670                0.000
  8                0.002                0.000
  9                0.000                0.000
 10                0.000               14.137
 11               20.418                0.000
 12                0.000                0.000
 13                0.000                3.327
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7219269       2805          0          0          0          0        933        933          0        939          0        589
  1   37816150      10006          0          0        269        269       4734       4734          0          0          0      43094
  2    6973157    6465448          0          0          0          0       1676       1676          0    6462096          0        570
  3   37723722    5287878          0          0        579        580       5326       5326          0    5276067          0        352
  4    3982759       1044          0          0          0          0        522        522          0          0          0        233
  5   36626106      10369          0          0        747        746       4438       4438          0          0          0        211
  6   11915365       7321          0          0          0          0       1544       1544          0       4233          0        403
  7   42135728      15066          0          0        653        653       5248       5248          0       3264          0        174
  8   10616770       2532          0          0          2          2       1264       1264          0          0          0        570
  9    3975235        966          0          0          0          0        483        483          0          0          0        497
 10    2072466        538          0          0          0          0        269        269          0          0          0        289
 11   40106886      10836          0          0        442        442       4976       4976          0          0          0        472
 12    6251481       1636          0          0          0          0        818        818          0          0          0        413
 13    2687755        706          0          0          0          0        353        353          0          0          0        303
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.205s (  883.854 blocks/s,  55.241 Mb/s)
Attachment 27: Screenshot_from_2022-05-15_00-56-27.png
Screenshot_from_2022-05-15_00-56-27.png
Attachment 28: Screenshot_from_2022-05-15_00-57-44.png
Screenshot_from_2022-05-15_00-57-44.png
Attachment 29: Screenshot_from_2022-05-15_00-59-12.png
Screenshot_from_2022-05-15_00-59-12.png
Attachment 30: Screenshot_from_2022-05-15_01-02-36.png
Screenshot_from_2022-05-15_01-02-36.png
Attachment 31: Screenshot_from_2022-05-15_01-04-44.png
Screenshot_from_2022-05-15_01-04-44.png
Attachment 32: Screenshot_from_2022-05-15_01-05-49.png
Screenshot_from_2022-05-15_01-05-49.png
Attachment 33: Screenshot_from_2022-05-15_01-13-22.png
Screenshot_from_2022-05-15_01-13-22.png
Attachment 34: Screenshot_from_2022-05-15_01-15-01.png
Screenshot_from_2022-05-15_01-15-01.png
Attachment 35: Screenshot_from_2022-05-15_04-13-03.png
Screenshot_from_2022-05-15_04-13-03.png
Attachment 36: Screenshot_from_2022-05-15_04-13-31.png
Screenshot_from_2022-05-15_04-13-31.png
Attachment 37: Screenshot_from_2022-05-15_04-13-52.png
Screenshot_from_2022-05-15_04-13-52.png
Attachment 38: Screenshot_from_2022-05-15_04-14-31.png
Screenshot_from_2022-05-15_04-14-31.png
Attachment 39: Screenshot_from_2022-05-15_07-05-19.png
Screenshot_from_2022-05-15_07-05-19.png
Attachment 40: Screenshot_from_2022-05-15_07-05-50.png
Screenshot_from_2022-05-15_07-05-50.png
Attachment 41: Screenshot_from_2022-05-15_07-06-15.png
Screenshot_from_2022-05-15_07-06-15.png
Attachment 42: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 43: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 44: Screenshot_from_2022-05-15_07-12-37.png
Screenshot_from_2022-05-15_07-12-37.png
Attachment 45: Screenshot_from_2022-05-15_07-14-49.png
Screenshot_from_2022-05-15_07-14-49.png
Attachment 46: Screenshot_from_2022-05-15_07-16-18.png
Screenshot_from_2022-05-15_07-16-18.png
Attachment 47: Screenshot_from_2022-05-15_07-20-47.png
Screenshot_from_2022-05-15_07-20-47.png
Attachment 48: Screenshot_from_2022-05-15_07-22-02.png
Screenshot_from_2022-05-15_07-22-02.png
  454   Sat May 14 21:34:11 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Rates_2022-05-14_22-32-44.png
Rates_2022-05-14_22-32-44.png
Attachment 2: Temp_2022-05-14_22-31-37.png
Temp_2022-05-14_22-31-37.png
Attachment 3: Leakg_2022-05-14_22-30-28.png
Leakg_2022-05-14_22-30-28.png
  453   Sat May 14 19:39:13 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Stat_2022-05-14_20-36-15.png
Stat_2022-05-14_20-36-15.png
Attachment 2: Temp_2022-05-14_20-35-19.png
Temp_2022-05-14_20-35-19.png
Attachment 3: Leakage_2022-05-14_20-33-32.png
Leakage_2022-05-14_20-33-32.png
  452   Sat May 14 17:34:40 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: Stat_2022-05-14_18-33-16.png
Stat_2022-05-14_18-33-16.png
Attachment 2: Temp_2022-05-14_18-32-07.png
Temp_2022-05-14_18-32-07.png
Attachment 3: Leakage_2022-05-14_18-30-52.png
Leakage_2022-05-14_18-30-52.png
  451   Sat May 14 15:35:24 2022 BA, AASaturday 14 May
Attachment 1: Stais_from_2022-05-14_16-30-53.png
Stais_from_2022-05-14_16-30-53.png
Attachment 2: Temp_2022-05-14_16-32-42.png
Temp_2022-05-14_16-32-42.png
Attachment 3: Leakage_2022-05-14_16-33-45.png
Leakage_2022-05-14_16-33-45.png
  450   Sat May 14 13:01:58 2022 MS, OHSaturday 14 May

14:00

Attachment 1: Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_14-03-54.png
Screenshot_from_2022-05-14_14-03-54.png
  449   Sat May 14 10:59:52 2022 MS, OHSaturday 14 May

12:00

Attachment 1: Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_12-01-54.png
Screenshot_from_2022-05-14_12-01-54.png
  448   Sat May 14 09:00:21 2022 MS, OHSaturday 14 May

10:00

Attachment 1: Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_10-02-51.png
Screenshot_from_2022-05-14_10-02-51.png
  447   Sat May 14 06:57:21 2022 MS, OHSaturday 14 May
08:00 Took over the shift from Philippos
      In system wide checks aida09 fails ASIC clock check but it is bit 6 which is ok
  446   Sat May 14 03:14:00 2022 PPShift Checks

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

 

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

 

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR


Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

 

Attachment 1: stats-220514-0428.png
stats-220514-0428.png
Attachment 2: temps-220514-0417.png
temps-220514-0417.png
Attachment 3: scalers-220514-0426.png
scalers-220514-0426.png
Attachment 4: I-V-220514-0431.png
I-V-220514-0431.png
Attachment 5: stats-220514-0639.png
stats-220514-0639.png
Attachment 6: temps-220514-0639.png
temps-220514-0639.png
Attachment 7: scalers-220514-0641.png
scalers-220514-0641.png
Attachment 8: I-V-220514-0640.png
I-V-220514-0640.png
  445   Sat May 14 01:52:03 2022 ML2h-Shift Checks

AIDA stats ok

Leakage current ok

Temperatires ok

grafana ok

ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.

System wide check done and same results as earlier:

aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_02-48-14.png
Stats_Screenshot_from_2022-05-14_02-48-14.png
Attachment 2: Temp_Screenshot_from_2022-05-14_02-47-31.png
Temp_Screenshot_from_2022-05-14_02-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-14_02-46-42.png
HV_LC_Screenshot_from_2022-05-14_02-46-42.png
Attachment 4: ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
Attachment 5: Grafana_Screenshot_from_2022-05-14_03-50-00.png
Grafana_Screenshot_from_2022-05-14_03-50-00.png
  444   Fri May 13 23:37:59 2022 ML2h-Shift Checks
Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_00-40-56.png
Stats_Screenshot_from_2022-05-14_00-40-56.png
Attachment 2: HV_LC_Screenshot_from_2022-05-14_00-39-51.png
HV_LC_Screenshot_from_2022-05-14_00-39-51.png
Attachment 3: temp_Screenshot_from_2022-05-14_00-41-44.png
temp_Screenshot_from_2022-05-14_00-41-44.png
Attachment 4: UCESB_Screenshot_from_2022-05-14_01-00-00.png
UCESB_Screenshot_from_2022-05-14_01-00-00.png
  443   Fri May 13 15:00:23 2022 CB13 March - 16:00-24:00 shift
16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before
Attachment 1: 1305_1600_ucesb.png
1305_1600_ucesb.png
Attachment 2: 1305_1600_StatsTemps.png
1305_1600_StatsTemps.png
Attachment 3: 1305_1600_Grafana.png
1305_1600_Grafana.png
Attachment 4: 1305_1800_StatsLeakage.png
1305_1800_StatsLeakage.png
Attachment 5: 1305_2030_Stats.png
1305_2030_Stats.png
Attachment 6: 1305_2030_ucesb.png
1305_2030_ucesb.png
Attachment 7: 1305_2030_Grafana.png
1305_2030_Grafana.png
Attachment 8: 1305_2300_StatsLeakage.png
1305_2300_StatsLeakage.png
Attachment 9: 1305_2300_ucesb.png
1305_2300_ucesb.png
Attachment 10: 1305_2300_Grafana.png
1305_2300_Grafana.png
  442   Fri May 13 07:36:18 2022 OH, TDFriday 13 May
08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)
Attachment 1: 220513_0839_R1_327_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259232212 ( 1719602.4 Hz)
        Other data format:    2687788 (   17829.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        986 (       6.5 Hz)
                               RESUME:        986 (       6.5 Hz)
                              SYNC100:      32562 (     216.0 Hz)
                              WR48-63:      32562 (     216.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2620692 (   17384.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     130683 (     866.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      150.751 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1                9.978                0.000
  2                0.059                0.000
  3               21.640                0.000
  4                0.158                0.000
  5               66.897                0.000
  6                1.778                0.000
  7               36.972                0.000
  8                0.172                0.000
  9                0.018                0.000
 10                0.000                7.315
 11               28.135                0.000
 12                0.007                0.000
 13                0.000                3.034
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7933827       2887          0          0          5          5        987        987          0        903          0       1141
  1   37526918       9508          0          0         50         50       4704       4704          0          0          0     122502
  2    8368319    1400718          0          0          2          2       1222       1222          0    1398270          0        647
  3   39185673    1207805          0          0        156        156       5116       5116          0    1197261          0        703
  4    4609566       1178          0          0          5          5        584        584          0          0          0       1170
  5   40049357      10540          0          0        309        309       4961       4961          0          0          0        297
  6   13162553      17502          0          0         13         13       1685       1685          0      14106          0        302
  7   41199857      20850          0          0        248        248       5101       5101          0      10152          0        241
  8   12021437       2864          0          0          5          5       1427       1427          0          0          0        467
  9    4748197       1188          0          0          2          2        592        592          0          0          0       1165
 10    3089949        744          0          0          0          0        372        372          0          0          0        247
 11   38258323       9856          0          0        190        190       4738       4738          0          0          0       1052
 12    6307578       1518          0          0          1          1        758        758          0          0          0        311
 13    2770658        630          0          0          0          0        315        315          0          0          0        438
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.600s (  952.392 blocks/s,  59.525 Mb/s)
Attachment 2: Screenshot_from_2022-05-13_14-28-00.png
Screenshot_from_2022-05-13_14-28-00.png
Attachment 3: Screenshot_from_2022-05-13_14-27-51.png
Screenshot_from_2022-05-13_14-27-51.png
Attachment 4: Screenshot_from_2022-05-13_14-27-31.png
Screenshot_from_2022-05-13_14-27-31.png
Attachment 5: Screenshot_from_2022-05-13_14-27-17.png
Screenshot_from_2022-05-13_14-27-17.png
Attachment 6: Screenshot_from_2022-05-13_14-26-32.png
Screenshot_from_2022-05-13_14-26-32.png
Attachment 7: Screenshot_from_2022-05-13_14-26-26.png
Screenshot_from_2022-05-13_14-26-26.png
Attachment 8: Screenshot_from_2022-05-13_14-26-21.png
Screenshot_from_2022-05-13_14-26-21.png
Attachment 9: Screenshot_from_2022-05-13_14-26-08.png
Screenshot_from_2022-05-13_14-26-08.png
Attachment 10: Screenshot_from_2022-05-13_14-25-46.png
Screenshot_from_2022-05-13_14-25-46.png
Attachment 11: Screenshot_from_2022-05-13_14-25-38.png
Screenshot_from_2022-05-13_14-25-38.png
Attachment 12: Screenshot_from_2022-05-13_14-25-22.png
Screenshot_from_2022-05-13_14-25-22.png
Attachment 13: Screenshot_from_2022-05-13_14-24-07.png
Screenshot_from_2022-05-13_14-24-07.png
Attachment 14: Screenshot_from_2022-05-13_14-22-44.png
Screenshot_from_2022-05-13_14-22-44.png
Attachment 15: Screenshot_from_2022-05-13_14-21-16.png
Screenshot_from_2022-05-13_14-21-16.png
Attachment 16: Screenshot_from_2022-05-13_14-21-04.png
Screenshot_from_2022-05-13_14-21-04.png
Attachment 17: Screenshot_from_2022-05-13_14-20-34.png
Screenshot_from_2022-05-13_14-20-34.png
Attachment 18: Screenshot_from_2022-05-13_14-20-14.png
Screenshot_from_2022-05-13_14-20-14.png
Attachment 19: Screenshot_from_2022-05-13_14-19-10.png
Screenshot_from_2022-05-13_14-19-10.png
Attachment 20: Screenshot_from_2022-05-13_14-18-40.png
Screenshot_from_2022-05-13_14-18-40.png
Attachment 21: Screenshot_from_2022-05-13_14-18-31.png
Screenshot_from_2022-05-13_14-18-31.png
Attachment 22: Screenshot_from_2022-05-13_14-18-14.png
Screenshot_from_2022-05-13_14-18-14.png
Attachment 23: Screenshot_from_2022-05-13_14-17-38.png
Screenshot_from_2022-05-13_14-17-38.png
Attachment 24: Screenshot_from_2022-05-13_14-17-23.png
Screenshot_from_2022-05-13_14-17-23.png
Attachment 25: Screenshot_from_2022-05-13_14-17-09.png
Screenshot_from_2022-05-13_14-17-09.png
Attachment 26: Screenshot_from_2022-05-13_14-16-56.png
Screenshot_from_2022-05-13_14-16-56.png
Attachment 27: Screenshot_from_2022-05-13_14-16-30.png
Screenshot_from_2022-05-13_14-16-30.png
Attachment 28: Screenshot_from_2022-05-13_14-16-11.png
Screenshot_from_2022-05-13_14-16-11.png
Attachment 29: R4_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259102731 ( 1572653.5 Hz)
        Other data format:    2817271 (   17099.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2412 (      14.6 Hz)
                               RESUME:       2414 (      14.7 Hz)
                              SYNC100:      32614 (     198.0 Hz)
                              WR48-63:      32614 (     198.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2747217 (   16674.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      30594 (     185.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      164.755 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                5.297                0.000
  2                0.000                0.000
  3               18.287                0.000
  4                0.000                0.000
  5               77.894                0.000
  6                0.000                0.000
  7               25.402                0.000
  8                0.001                0.000
  9                0.000                0.000
 10                0.000                2.705
 11               15.617                0.000
 12                0.000                0.000
 13                0.000               29.550
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6844719       2775          0          0          0          0        894        894          0        987          0        375
  1   39920437      10378          0          0        277        277       4912       4912          0          0          0      27546
  2    7063505    1453891          0          0          0          0       1085       1085          0    1451721          0        328
  3   42633132    1301928          0          0        410        410       5590       5590          0    1289928          0        251
  4    3625899        900          0          0          0          0        450        450          0          0          0        125
  5   38867835      10853          0          0        768        769       4658       4658          0          0          0        129
  6    7590830       4334          0          0          0          0        934        934          0       2466          0        227
  7   44498336      14120          0          0        564        565       5438       5438          0       2115          0        113
  8   10949373       2716          0          0          1          1       1357       1357          0          0          0        345
  9    3887105        996          0          0          0          0        498        498          0          0          0        273
 10    1999345        532          0          0          0          0        266        266          0          0          0        160
 11   43113077      11754          0          0        392        392       5485       5485          0          0          0        288
 12    6030564       1596          0          0          0          0        798        798          0          0          0        252
 13    2078574        498          0          0          0          0        249        249          0          0          0        182
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
Attachment 30: Screenshot_from_2022-05-13_15-11-36.png
Screenshot_from_2022-05-13_15-11-36.png
Attachment 31: Screenshot_from_2022-05-13_15-11-29.png
Screenshot_from_2022-05-13_15-11-29.png
Attachment 32: Screenshot_from_2022-05-13_15-11-17.png
Screenshot_from_2022-05-13_15-11-17.png
Attachment 33: Screenshot_from_2022-05-13_15-10-55.png
Screenshot_from_2022-05-13_15-10-55.png
Attachment 34: Screenshot_from_2022-05-13_15-10-41.png
Screenshot_from_2022-05-13_15-10-41.png
Attachment 35: Screenshot_from_2022-05-13_15-10-30.png
Screenshot_from_2022-05-13_15-10-30.png
  441   Fri May 13 06:56:14 2022 OH, BA, MALow Rates

 

Quote:

The rate is very low in all aida, not sure .

This was because the statistics page had been changed to transfer buffers. The rates were ok

  440   Fri May 13 06:47:33 2022 ML2h-Shift Checks

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

Attachment 1: Stats_Screenshot_from_2022-05-13_07-43-54.png
Stats_Screenshot_from_2022-05-13_07-43-54.png
Attachment 2: Temp_Screenshot_from_2022-05-13_07-43-13.png
Temp_Screenshot_from_2022-05-13_07-43-13.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_07-42-38.png
HV_LC_Screenshot_from_2022-05-13_07-42-38.png
  439   Fri May 13 04:45:23 2022 ML2h-Shift Checks

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

Attachment 1: Stats_Screenshot_from_2022-05-13_05-43-14.png
Stats_Screenshot_from_2022-05-13_05-43-14.png
Attachment 2: Temp_Screenshot_from_2022-05-13_05-42-10.png
Temp_Screenshot_from_2022-05-13_05-42-10.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_05-41-23.png
HV_LC_Screenshot_from_2022-05-13_05-41-23.png
  438   Fri May 13 02:56:36 2022 ML2h-Shift Checks

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

Attachment 1: Stats_Screenshot_from_2022-05-13_03-43-34.png
Stats_Screenshot_from_2022-05-13_03-43-34.png
Attachment 2: Temp_Screenshot_from_2022-05-13_03-44-41.png
Temp_Screenshot_from_2022-05-13_03-44-41.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_03-41-28.png
HV_LC_Screenshot_from_2022-05-13_03-41-28.png
  437   Fri May 13 02:28:36 2022 MLbeam dropped

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

  436   Fri May 13 00:50:16 2022 ML2h-Shift Checks

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

Attachment 1: Stats_Screenshot_from_2022-05-13_01-45-56.png
Stats_Screenshot_from_2022-05-13_01-45-56.png
Attachment 2: Temp_Screenshot_from_2022-05-13_01-47-31.png
Temp_Screenshot_from_2022-05-13_01-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_01-48-21.png
HV_LC_Screenshot_from_2022-05-13_01-48-21.png
  435   Thu May 12 22:49:39 2022 BA, MALow Rates

The rate is very low in all aida, not sure .

Attachment 1: RatesLow2022-05-12_23-48-01.png
RatesLow2022-05-12_23-48-01.png
  434   Thu May 12 21:43:49 2022 BA, MARates

 

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

 

Attachment 1: Stati2022-05-12_22-47-26.png
Stati2022-05-12_22-47-26.png
Attachment 2: Temp2022-05-12_22-48-41.png
Temp2022-05-12_22-48-41.png
Attachment 3: LeakageCurrent2022-05-12_22-45-44.png
LeakageCurrent2022-05-12_22-45-44.png
  433   Thu May 12 16:30:47 2022 NH, OH, TDNotes for mounting a second switch
Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida*
  432   Thu May 12 10:10:51 2022 TD, OHAIDA data throughput
Attachment 1 - top aida06

Attachment 2 - top aida01

Attachment 3 - top aida-gsi (AIDA server)

Attachment 4 - TapeServer

Attachment 5 - NewMerger stats block 6 (aida06)

Attachment 6 - NewMerger stats block 1 (aida01)

Attachment 7 - NewMerger stats block 0 (all FEE64s)

Attachment 8 - NewMerger options

Attachment 9 - NewMerger link stats

Attachment 10 - NewMerger

Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats

Attacgment 15 - Network traffic
Attachment 1: Screenshot_from_2022-05-12_11-08-34.png
Screenshot_from_2022-05-12_11-08-34.png
Attachment 2: Screenshot_from_2022-05-12_11-07-44.png
Screenshot_from_2022-05-12_11-07-44.png
Attachment 3: Screenshot_from_2022-05-12_11-07-14.png
Screenshot_from_2022-05-12_11-07-14.png
Attachment 4: Screenshot_from_2022-05-12_11-05-53.png
Screenshot_from_2022-05-12_11-05-53.png
Attachment 5: Screenshot_from_2022-05-12_11-05-40.png
Screenshot_from_2022-05-12_11-05-40.png
Attachment 6: Screenshot_from_2022-05-12_11-05-18.png
Screenshot_from_2022-05-12_11-05-18.png
Attachment 7: Screenshot_from_2022-05-12_11-05-09.png
Screenshot_from_2022-05-12_11-05-09.png
Attachment 8: Screenshot_from_2022-05-12_11-04-46.png
Screenshot_from_2022-05-12_11-04-46.png
Attachment 9: Screenshot_from_2022-05-12_11-04-22.png
Screenshot_from_2022-05-12_11-04-22.png
Attachment 10: Screenshot_from_2022-05-12_11-03-49.png
Screenshot_from_2022-05-12_11-03-49.png
Attachment 11: Screenshot_from_2022-05-12_11-03-37.png
Screenshot_from_2022-05-12_11-03-37.png
Attachment 12: Screenshot_from_2022-05-12_11-03-28.png
Screenshot_from_2022-05-12_11-03-28.png
Attachment 13: Screenshot_from_2022-05-12_11-03-13.png
Screenshot_from_2022-05-12_11-03-13.png
Attachment 14: Screenshot_from_2022-05-12_11-02-58.png
Screenshot_from_2022-05-12_11-02-58.png
Attachment 15: Screenshot_from_2022-05-12_11-15-56.png
Screenshot_from_2022-05-12_11-15-56.png
  431   Thu May 12 07:09:43 2022 TDThursday 12 May 08:00-24:00
08.10 AIDA DAQ status
    
      14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
      AD9222 OFF - waveforms disabed
      disc - all disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23

      BNC PB-5
      amplitude 1.000V
      attenuation x1
      tau_d 1ms (tail pulse)
      polarity +
      frequency 22Hz

08.14 all histograms zero'd
2
08.15 system wide check counter baseline

08.17 check ASIC control - all FEE64s all ASICs

Attachment 1 - grafana DSSSD bias & leakage current - OK

Attachment 2 - per FEE64 stat spectra

Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips

Attachments 5 & 6 - per FEE64 1.8.L spectra
 pulser peak width aida01 124ch FWHM, aida04 293ch FWHM

Attachment 7 - per FEE64 rate spectra

Attachments 8-12 - system wide checks - all OK

Attachments 13-16 - statistics - adc, pause, resume & correlation data

Attachment 17 - FEE64 temps OK

Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends

Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages

08.44 DAQ continues - file NOTAPE/R18 - no storage mode

08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc

08.53 no storage mode enabled

Attachment 21 - analysis of file NOTAPE/R18_20

Attachment 22- analysis of file NOTAPE/R19_19
               AIDA12 slow comp up to 0xd improved deadtime drastically for FEE

10:54 Beam on detector. Take another run to test deadtime - attachment 23
      Deadtimes ranging from 10-40% in n+n FEEs

Attachment 23 & 24 aida06 aida02 1.*.H spectra


14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
        Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime)
Attachment 1: Screenshot_from_2022-05-12_08-25-44.png
Screenshot_from_2022-05-12_08-25-44.png
Attachment 2: Screenshot_from_2022-05-12_08-25-20.png
Screenshot_from_2022-05-12_08-25-20.png
Attachment 3: Screenshot_from_2022-05-12_08-24-01.png
Screenshot_from_2022-05-12_08-24-01.png
Attachment 4: Screenshot_from_2022-05-12_08-23-35.png
Screenshot_from_2022-05-12_08-23-35.png
Attachment 5: Screenshot_from_2022-05-12_08-22-24.png
Screenshot_from_2022-05-12_08-22-24.png
Attachment 6: Screenshot_from_2022-05-12_08-21-18.png
Screenshot_from_2022-05-12_08-21-18.png
Attachment 7: Screenshot_from_2022-05-12_08-19-15.png
Screenshot_from_2022-05-12_08-19-15.png
Attachment 8: Screenshot_from_2022-05-12_08-16-40.png
Screenshot_from_2022-05-12_08-16-40.png
Attachment 9: Screenshot_from_2022-05-12_08-16-30.png
Screenshot_from_2022-05-12_08-16-30.png
Attachment 10: Screenshot_from_2022-05-12_08-16-25.png
Screenshot_from_2022-05-12_08-16-25.png
Attachment 11: Screenshot_from_2022-05-12_08-16-17.png
Screenshot_from_2022-05-12_08-16-17.png
Attachment 12: Screenshot_from_2022-05-12_08-16-08.png
Screenshot_from_2022-05-12_08-16-08.png
Attachment 13: Screenshot_from_2022-05-12_08-10-02.png
Screenshot_from_2022-05-12_08-10-02.png
Attachment 14: Screenshot_from_2022-05-12_08-09-44.png
Screenshot_from_2022-05-12_08-09-44.png
Attachment 15: Screenshot_from_2022-05-12_08-09-20.png
Screenshot_from_2022-05-12_08-09-20.png
Attachment 16: Screenshot_from_2022-05-12_08-09-02.png
Screenshot_from_2022-05-12_08-09-02.png
Attachment 17: Screenshot_from_2022-05-12_08-08-39.png
Screenshot_from_2022-05-12_08-08-39.png
Attachment 18: Screenshot_from_2022-05-12_08-08-26.png
Screenshot_from_2022-05-12_08-08-26.png
Attachment 19: Screenshot_from_2022-05-12_08-39-14.png
Screenshot_from_2022-05-12_08-39-14.png
Attachment 20: Screenshot_from_2022-05-12_08-39-05.png
Screenshot_from_2022-05-12_08-39-05.png
Attachment 21: R18_20
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259331519 ( 1693112.6 Hz)
        Other data format:    2588481 (   16899.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1016 (       6.6 Hz)
                               RESUME:       1017 (       6.6 Hz)
                              SYNC100:      32570 (     212.6 Hz)
                              WR48-63:      32570 (     212.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2521308 (   16461.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.169 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.171                0.000
  1                9.417                0.000
  2                0.061                0.000
  3               24.348                0.000
  4                0.008                0.000
  5               72.336                0.000
  6                0.231                0.000
  7               25.570                0.000
  8                0.125                0.000
  9                0.007                0.000
 10                0.000                7.342
 11               41.574                0.000
 12                0.027                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7431923      12514          0          0          8          8        933        933          0      10632          0          0
  1   37380104       9380          0          0         51         51       4639       4639          0          0          0          0
  2    7217279    1365495          0          0          3          3       1056       1056          0    1363377          0          0
  3   38015689    1157497          0          0        179        179       4920       4920          0    1147299          0          0
  4    4440445       1078          0          0          2          2        537        537          0          0          0          0
  5   40380917      10697          0          0        319        320       5029       5029          0          0          0          0
  6   14913071       3788          0          0          5          5       1889       1889          0          0          0          0
  7   41232478      10490          0          0        181        181       5064       5064          0          0          0          0
  8   11952709       2932          0          0          4          4       1462       1462          0          0          0          0
  9    4910914       1250          0          0          1          1        624        624          0          0          0          0
 10    2469929        580          0          0          0          0        290        290          0          0          0          0
 11   39767127      10486          0          0        257        257       4986       4986          0          0          0          0
 12    5557988       1406          0          0          6          6        697        697          0          0          0          0
 13    3660946        888          0          0          0          0        444        444          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.578s (  874.840 blocks/s,  54.677 Mb/s)
Attachment 22: R19_19.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      17139
          ADC data format:  138519076 ( 1563611.9 Hz)
        Other data format:    1763639 (   19908.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        358 (       4.0 Hz)
                               RESUME:        358 (       4.0 Hz)
                              SYNC100:      17469 (     197.2 Hz)
                              WR48-63:      17469 (     197.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1727985 (   19505.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       88.589 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.084                0.000
  1                0.528                0.000
  2                0.008                0.000
  3               10.973                0.000
  4                0.000                0.000
  5               30.421                0.000
  6                0.000                0.000
  7                7.252                0.000
  8                0.055                0.000
  9                0.000                0.000
 10                0.000               12.541
 11                3.332                0.000
 12                0.009                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4390551       7288          0          0          4          4        565        565          0       6150          0          0
  1   21021919       5186          0          0          9          9       2584       2584          0          0          0          0
  2    3174721     918030          0          0          1          1        518        518          0     916992          0          0
  3   22097261     810837          0          0         77         77       2920       2920          0     804843          0          0
  4    2596723        620          0          0          0          0        310        310          0          0          0          0
  5   23543560       6240          0          0        174        174       2946       2946          0          0          0          0
  6    4211356       1020          0          0          0          0        510        510          0          0          0          0
  7   24080430       6296          0          0         63         63       3085       3085          0          0          0          0
  8    4823696       1068          0          0          3          3        531        531          0          0          0          0
  9    1770702        408          0          0          0          0        204        204          0          0          0          0
 10     933213        222          0          0          0          0        111        111          0          0          0          0
 11   21419475       5280          0          0         21         21       2619       2619          0          0          0          0
 12    2929744        698          0          0          6          6        343        343          0          0          0          0
 13    1525725        446          0          0          0          0        223        223          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   17.484s (  980.247 blocks/s,  61.265 Mb/s)
Attachment 23: R20_20.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       4909
          ADC data format:   39765494 ( 1644562.3 Hz)
        Other data format:     414671 (   17149.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        151 (       6.2 Hz)
                               RESUME:        152 (       6.3 Hz)
                              SYNC100:       4999 (     206.7 Hz)
                              WR48-63:       4999 (     206.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     404370 (   16723.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      23252 (     961.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       24.180 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.954                0.000
  2                0.000                0.000
  3                4.569                0.000
  4                0.000                0.000
  5               10.824                0.000
  6                0.000                0.000
  7                5.869                0.000
  8                0.000                0.000
  9                0.008                0.000
 10                0.000                0.000
 11                2.942                0.000
 12                0.007                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    1157062       1976          0          0          0          0        148        148          0       1680          0        159
  1    5834981       1480          0          0         12         12        728        728          0          0          0      22412
  2    1125294     220409          0          0          0          0        178        178          0     220053          0         83
  3    5924210     180469          0          0         32         32        741        741          0     178923          0         86
  4     669291        180          0          0          0          0         90         90          0          0          0         10
  5    6069826       1577          0          0         47         48        741        741          0          0          0         20
  6    1749760       2346          0          0          0          0        222        222          0       1902          0         32
  7    6604292       3590          0          0         38         38        851        851          0       1812          0         32
  8    1794890        416          0          0          0          0        208        208          0          0          0         42
  9     736012        164          0          0          1          1         81         81          0          0          0        124
 10     410664         98          0          0          0          0         49         49          0          0          0         15
 11    6315369       1576          0          0         19         19        769        769          0          0          0        153
 12     852659        248          0          0          2          2        122        122          0          0          0         23
 13     521184        142          0          0          0          0         71         71          0          0          0         61
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    5.102s (  962.254 blocks/s,  60.141 Mb/s)
Attachment 24: Screenshot_from_2022-05-12_11-57-44.png
Screenshot_from_2022-05-12_11-57-44.png
Attachment 25: Screenshot_from_2022-05-12_11-56-42.png
Screenshot_from_2022-05-12_11-56-42.png
Attachment 26: Screenshot_from_2022-05-12_22-32-53.png
Screenshot_from_2022-05-12_22-32-53.png
Attachment 27: Screenshot_from_2022-05-12_22-32-22.png
Screenshot_from_2022-05-12_22-32-22.png
  430   Thu May 12 00:37:52 2022 OH, AMThursday 12 May
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
      That is now complete.
      Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc

      R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
      Will lower 2 and 4 to 200 and raise 6 and 8 to 300

      R17 2 good, 4 shakey 6 and 8 poor - attachment 2

      R18 2 and 4 at 220 and 6 and 7 at 350
      2 and 8 low deadtime, 4 and 6 could be better. - attachment 3


      Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
      Statistics at this rate - attachment 4




02:18 Statistics - attachment 7
      Temperatures - attachment 6
      Bias & leakage current - attachment 5

      Clock Status - OK
      White Rabbit - OK
      The other 3 checks currently aren't working




04:11 Statistics - attachment 10
      Temperatures - attachment 9
      Bias & leakage current - attachment 8

      Clock Status - OK
      White Rabbit - OK




06:05 Statistics - attachment 13
      Temperatures - attachment 12
      Bias & leakage current - attachment 11

      Clock Status - OK
      White Rabbit - OK
Attachment 1: R16_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      18197
          ADC data format:  144240941 ( 1037416.0 Hz)
        Other data format:    4701506 (   33814.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        651 (       4.7 Hz)
                               RESUME:        651 (       4.7 Hz)
                              SYNC100:      18715 (     134.6 Hz)
                              WR48-63:      18715 (     134.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4662774 (   33535.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       9906 (      71.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      139.039 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               96.976
  1                1.224                0.000
  2                0.000                2.784
  3                4.445                0.000
  4                0.000              108.587
  5               78.372                0.000
  6                0.000               48.067
  7               67.687                0.000
  8                0.000              100.043
  9                0.000              102.969
 10                0.000              122.226
 11                0.000               21.170
 12                0.000                2.713
 13                0.000              116.657
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     346857       9766          0          0          0          0         47         47          0       9672          0       2994
  1   28465912       7268          0          0         10         10       3624       3624          0          0          0       3890
  2     235746    2352664          0          0          0          0        299        299          0    2352066          0        528
  3   33196911    2283279          0          0         20         20       4471       4471          0    2274297          0        644
  4     346134         72          0          0          0          0         36         36          0          0          0          4
  5   37750759       9970          0          0        309        309       4676       4676          0          0          0        836
  6     855292      17907          0          0          0          0        114        114          0      17679          0        522
  7   38520306      19374          0          0        312        312       4845       4845          0       9060          0        313
  8     312768         68          0          0          0          0         34         34          0          0          0         23
  9     342945         90          0          0          0          0         45         45          0          0          0         14
 10     176833         46          0          0          0          0         23         23          0          0          0          7
 11    1559416        390          0          0          0          0        195        195          0          0          0         62
 12    1848830        538          0          0          0          0        269        269          0          0          0         19
 13     282232         74          0          0          0          0         37         37          0          0          0         50
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   18.576s (  979.614 blocks/s,  61.226 Mb/s)
Attachment 2: R17_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9844
          ADC data format:   78473161 ( 1119546.3 Hz)
        Other data format:    2099979 (   29959.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        470 (       6.7 Hz)
                               RESUME:        470 (       6.7 Hz)
                              SYNC100:      10105 (     144.2 Hz)
                              WR48-63:      10105 (     144.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2078829 (   29657.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       70.094 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               61.805
  1                4.886                0.000
  2                0.000                0.000
  3               17.776                0.000
  4                0.000               55.325
  5               34.711                0.000
  6                0.000               20.226
  7               27.353                0.000
  8                0.000               57.388
  9                0.000               53.302
 10                0.000               59.261
 11                0.000                7.690
 12                0.000               11.252
 13                0.000               58.364
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     137192       4907          0          0          0          0         16         16          0       4875          0          0
  1   18540829       4668          0          0         40         40       2294       2294          0          0          0          0
  2     107308    1183296          0          0          0          0        177        177          0    1182942          0          0
  3   18083643     896022          0          0        126        126       2379       2379          0     891012          0          0
  4     171162         40          0          0          0          0         20         20          0          0          0          0
  5   19343787       5154          0          0        155        155       2422       2422          0          0          0          0
  6     395908        126          0          0          0          0         63         63          0          0          0          0
  7   19401210       5206          0          0        149        149       2454       2454          0          0          0          0
  8     159836         40          0          0          0          0         20         20          0          0          0          0
  9     172050         38          0          0          0          0         19         19          0          0          0          0
 10      94357         20          0          0          0          0         10         10          0          0          0          0
 11     792883        200          0          0          0          0        100        100          0          0          0          0
 12     933311        234          0          0          0          0        117        117          0          0          0          0
 13     139685         28          0          0          0          0         14         14          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.115s (  973.186 blocks/s,  60.824 Mb/s)
Attachment 3: R18_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9700
          ADC data format:   70825654 ( 1030660.0 Hz)
        Other data format:    8568846 (  124694.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        268 (       3.9 Hz)
                               RESUME:        268 (       3.9 Hz)
                              SYNC100:       9956 (     144.9 Hz)
                              WR48-63:       9956 (     144.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8548398 (  124396.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       68.719 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.039
  1                0.526                0.000
  2                0.044                0.000
  3               16.252                0.000
  4                0.000               41.343
  5               16.542                0.000
  6                0.000               37.115
  7                4.958                0.000
  8                0.000               62.160
  9                0.000               48.394
 10                0.000               57.328
 11                0.000               39.469
 12                0.000                0.000
 13                0.000               54.952
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     123341       4797          0          0          0          0          9          9          0       4779          0          0
  1   16458296       4022          0          0          5          5       2006       2006          0          0          0          0
  2      67098    4844172          0          0          2          2        649        649          0    4842870          0          2
  3   14877427    3705657          0          0        108        108       2346       2346          0    3700749          0          0
  4     201730         54          0          0          0          0         27         27          0          0          0          0
  5   18757776       4968          0          0        118        118       2366       2366          0          0          0          0
  6     275379         62          0          0          0          0         31         31          0          0          0          0
  7   18591077       4758          0          0         35         35       2344       2344          0          0          0          0
  8      87837         12          0          0          0          0          6          6          0          0          0          0
  9     156274         28          0          0          0          0         14         14          0          0          0          0
 10      61114         18          0          0          0          0          9          9          0          0          0          0
 11     292464         64          0          0          0          0         32         32          0          0          0          0
 12     736144        204          0          0          0          0        102        102          0          0          0          0
 13     139697         30          0          0          0          0         15         15          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.125s (  958.025 blocks/s,  59.877 Mb/s)
Attachment 4: 220512_0155_stats.png
220512_0155_stats.png
Attachment 5: 220512_0217_biasleakage.png
220512_0217_biasleakage.png
Attachment 6: 220512_0215_temps.png
220512_0215_temps.png
Attachment 7: 220512_0214_stats.png
220512_0214_stats.png
Attachment 8: 220512_0410_biasLeak.png
220512_0410_biasLeak.png
Attachment 9: 220512_0410_temps.png
220512_0410_temps.png
Attachment 10: 220512_0408_stats.png
220512_0408_stats.png
Attachment 11: 220512_0604_biasLeakage.png
220512_0604_biasLeakage.png
Attachment 12: 220512_0603_temps.png
220512_0603_temps.png
Attachment 13: 220512_0601_stats.png
220512_0601_stats.png
  429   Wed May 11 19:08:41 2022 CB, PPWednesday 11th May - evening shift

20:19 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

 

22:12 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

Attachment 1: 16.png
16.png
Attachment 2: 22.png
22.png
Attachment 3: 13.png
13.png
Attachment 4: 14.png
14.png
  428   Wed May 11 07:04:13 2022 OH, NH, TDWednesday 11th May
08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14
Attachment 1: 220511_0806_aida14.png
220511_0806_aida14.png
Attachment 2: 220511_0817_aida14_top.png
220511_0817_aida14_top.png
Attachment 3: Screenshot_from_2022-05-11_13-11-09.png
Screenshot_from_2022-05-11_13-11-09.png
Attachment 4: Screenshot_from_2022-05-11_13-10-29.png
Screenshot_from_2022-05-11_13-10-29.png
Attachment 5: Screenshot_from_2022-05-11_13-09-12.png
Screenshot_from_2022-05-11_13-09-12.png
Attachment 6: Screenshot_from_2022-05-11_13-16-32.png
Screenshot_from_2022-05-11_13-16-32.png
Attachment 7: Screenshot_from_2022-05-11_13-16-24.png
Screenshot_from_2022-05-11_13-16-24.png
Attachment 8: Screenshot_from_2022-05-11_13-16-01.png
Screenshot_from_2022-05-11_13-16-01.png
Attachment 9: Screenshot_from_2022-05-11_13-15-45.png
Screenshot_from_2022-05-11_13-15-45.png
  427   Tue May 10 07:41:28 2022 NH TDTuesday 10 May
8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam

Reseat aida05 ribbon cable on adapter board while in...

No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty

Noise level still high, threshold set to 0x20 (320 keV) which has decent rate

Start forwarding to MBS with 0x20 thresholds

10.12 slow comparator 0x20 -> 0xa

Attachments 1-5

Attachment 6 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 7 FEE64 temps OK

Attachment 8 DSSSD bias & leakage currents OK


13.30 system checks

      N.B. aida05 ASIC temp low and pause/resume data rate high

Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK

Attachments 10-15 spectra

Attacahments 16-19 system wide checks OK *except* WR error counter

Attachment 20 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 21 FEE64 temps OK

Attachment 22 DSSSD bias & leakage currents OK

Attachments 23-29 newMerger stats

20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable

22:26 - No implantation test although we had some brief beam through AIDA.
      - FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
        Will keep DAQ on 0x64 to eliminate deadtime issues for implantation
Attachment 1: Screenshot_from_2022-05-10_10-11-54.png
Screenshot_from_2022-05-10_10-11-54.png
Attachment 2: Screenshot_from_2022-05-10_10-10-09.png
Screenshot_from_2022-05-10_10-10-09.png
Attachment 3: Screenshot_from_2022-05-10_10-08-55.png
Screenshot_from_2022-05-10_10-08-55.png
Attachment 4: Screenshot_from_2022-05-10_10-08-02.png
Screenshot_from_2022-05-10_10-08-02.png
Attachment 5: Screenshot_from_2022-05-10_10-07-01.png
Screenshot_from_2022-05-10_10-07-01.png
Attachment 6: Screenshot_from_2022-05-10_10-06-30.png
Screenshot_from_2022-05-10_10-06-30.png
Attachment 7: Screenshot_from_2022-05-10_10-06-20.png
Screenshot_from_2022-05-10_10-06-20.png
Attachment 8: Screenshot_from_2022-05-10_10-06-15.png
Screenshot_from_2022-05-10_10-06-15.png
Attachment 9: Screenshot_from_2022-05-10_13-40-43.png
Screenshot_from_2022-05-10_13-40-43.png
Attachment 10: Screenshot_from_2022-05-10_13-40-01.png
Screenshot_from_2022-05-10_13-40-01.png
Attachment 11: Screenshot_from_2022-05-10_13-37-40.png
Screenshot_from_2022-05-10_13-37-40.png
Attachment 12: Screenshot_from_2022-05-10_13-37-33.png
Screenshot_from_2022-05-10_13-37-33.png
Attachment 13: Screenshot_from_2022-05-10_13-36-39.png
Screenshot_from_2022-05-10_13-36-39.png
Attachment 14: Screenshot_from_2022-05-10_13-35-06.png
Screenshot_from_2022-05-10_13-35-06.png
Attachment 15: Screenshot_from_2022-05-10_13-34-44.png
Screenshot_from_2022-05-10_13-34-44.png
Attachment 16: Screenshot_from_2022-05-10_13-31-05.png
Screenshot_from_2022-05-10_13-31-05.png
Attachment 17: Screenshot_from_2022-05-10_13-30-59.png
Screenshot_from_2022-05-10_13-30-59.png
Attachment 18: Screenshot_from_2022-05-10_13-30-52.png
Screenshot_from_2022-05-10_13-30-52.png
Attachment 19: Screenshot_from_2022-05-10_13-29-58.png
Screenshot_from_2022-05-10_13-29-58.png
Attachment 20: Screenshot_from_2022-05-10_13-29-42.png
Screenshot_from_2022-05-10_13-29-42.png
Attachment 21: Screenshot_from_2022-05-10_13-29-20.png
Screenshot_from_2022-05-10_13-29-20.png
Attachment 22: Screenshot_from_2022-05-10_13-28-48.png
Screenshot_from_2022-05-10_13-28-48.png
Attachment 23: Screenshot_from_2022-05-10_13-51-23.png
Screenshot_from_2022-05-10_13-51-23.png
Attachment 24: Screenshot_from_2022-05-10_13-51-05.png
Screenshot_from_2022-05-10_13-51-05.png
Attachment 25: Screenshot_from_2022-05-10_13-49-53.png
Screenshot_from_2022-05-10_13-49-53.png
Attachment 26: Screenshot_from_2022-05-10_13-49-41.png
Screenshot_from_2022-05-10_13-49-41.png
Attachment 27: Screenshot_from_2022-05-10_13-49-32.png
Screenshot_from_2022-05-10_13-49-32.png
Attachment 28: Screenshot_from_2022-05-10_13-49-12.png
Screenshot_from_2022-05-10_13-49-12.png
Attachment 29: Screenshot_from_2022-05-10_13-48-56.png
Screenshot_from_2022-05-10_13-48-56.png
  426   Mon May 9 08:36:23 2022 TD, NHMonday 9 May
09.32 Re-start AIDA

Attachments 1 & 2 1.8.W spectra 20us FSR

Attachments 3 & 4 1.8.L spectra
 aida10 98ch FWHM
 aida06 384 ch FWHM

Attachment 5 per FEE64 rate spectra

Attachments 6-13 system wide checks - all OK

Attachment 14 ADC data item stats
 n+n FEE64s c. 250k -> c. 370k
 aida01, 7 & 12 c. 200-400k 

Attachment 15 FEE64 temps OK

Attachment 16 DSSD bias & leakage current OK

Attachment 17 per FEE64 sys console status summary



10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
      ASIC check load

Attachments 18 ADC data item stats
 no change to n+n FEE64s, very high p+n FEE64s now at lower rates

Attachments 19 & 20 1.8.W spectra 20us FSR


18.27 all ADC data rates very high
      aida05, asic #1 is again producing HEC data
      aida14, asic #2 producing very little data

Attachments 23 & 24 1.8.W spectra 20us FSR

Attachments 25 & 26 1.8.L spectra
 aida10 165ch FWHM

Attachment 27 per FEE64 rate spectra

Attachment 28 ADC data item stats

Attachment 29 FEE64 temps OK

Attachment 30 DSSD bias & leakage current OK

Attachments 21 & 22 1.8.L spectra
 aida10 65ch FWHM
 aida06 394ch FWHM

Attachment 31 Rate with threesholds at 0x64

14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change. 


14.30 Replace FEE64 aida05
      old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
      old dhcpd.conf file backed up
Attachment 1: Screenshot_from_2022-05-09_09-58-10.png
Screenshot_from_2022-05-09_09-58-10.png
Attachment 2: Screenshot_from_2022-05-09_09-56-53.png
Screenshot_from_2022-05-09_09-56-53.png
Attachment 3: Screenshot_from_2022-05-09_09-54-56.png
Screenshot_from_2022-05-09_09-54-56.png
Attachment 4: Screenshot_from_2022-05-09_09-53-43.png
Screenshot_from_2022-05-09_09-53-43.png
Attachment 5: Screenshot_from_2022-05-09_09-51-41.png
Screenshot_from_2022-05-09_09-51-41.png
Attachment 6: Screenshot_from_2022-05-09_09-50-11.png
Screenshot_from_2022-05-09_09-50-11.png
Attachment 7: Screenshot_from_2022-05-09_09-50-07.png
Screenshot_from_2022-05-09_09-50-07.png
Attachment 8: Screenshot_from_2022-05-09_09-50-02.png
Screenshot_from_2022-05-09_09-50-02.png
Attachment 9: Screenshot_from_2022-05-09_09-48-13.png
Screenshot_from_2022-05-09_09-48-13.png
Attachment 10: Screenshot_from_2022-05-09_09-48-06.png
Screenshot_from_2022-05-09_09-48-06.png
Attachment 11: Screenshot_from_2022-05-09_09-48-00.png
Screenshot_from_2022-05-09_09-48-00.png
Attachment 12: Screenshot_from_2022-05-09_09-47-50.png
Screenshot_from_2022-05-09_09-47-50.png
Attachment 13: Screenshot_from_2022-05-09_09-47-27.png
Screenshot_from_2022-05-09_09-47-27.png
Attachment 14: Screenshot_from_2022-05-09_09-47-21.png
Screenshot_from_2022-05-09_09-47-21.png
Attachment 15: Screenshot_from_2022-05-09_09-47-12.png
Screenshot_from_2022-05-09_09-47-12.png
Attachment 16: Screenshot_from_2022-05-09_09-47-00.png
Screenshot_from_2022-05-09_09-47-00.png
Attachment 17: Screenshot_from_2022-05-09_09-36-29.png
Screenshot_from_2022-05-09_09-36-29.png
Attachment 18: Screenshot_from_2022-05-09_10-16-16.png
Screenshot_from_2022-05-09_10-16-16.png
Attachment 19: Screenshot_from_2022-05-09_10-17-40.png
Screenshot_from_2022-05-09_10-17-40.png
Attachment 20: Screenshot_from_2022-05-09_10-17-09.png
Screenshot_from_2022-05-09_10-17-09.png
Attachment 21: Screenshot_from_2022-05-09_10-32-52.png
Screenshot_from_2022-05-09_10-32-52.png
Attachment 22: Screenshot_from_2022-05-09_10-31-48.png
Screenshot_from_2022-05-09_10-31-48.png
Attachment 23: Screenshot_from_2022-05-09_18-16-39.png
Screenshot_from_2022-05-09_18-16-39.png
Attachment 24: Screenshot_from_2022-05-09_18-16-01.png
Screenshot_from_2022-05-09_18-16-01.png
Attachment 25: Screenshot_from_2022-05-09_18-15-32.png
Screenshot_from_2022-05-09_18-15-32.png
Attachment 26: Screenshot_from_2022-05-09_18-14-56.png
Screenshot_from_2022-05-09_18-14-56.png
Attachment 27: Screenshot_from_2022-05-09_18-13-43.png
Screenshot_from_2022-05-09_18-13-43.png
Attachment 28: Screenshot_from_2022-05-09_18-13-26.png
Screenshot_from_2022-05-09_18-13-26.png
Attachment 29: Screenshot_from_2022-05-09_18-13-16.png
Screenshot_from_2022-05-09_18-13-16.png
Attachment 30: Screenshot_from_2022-05-09_18-13-03.png
Screenshot_from_2022-05-09_18-13-03.png
Attachment 31: AnyDeskMSI_2022-05-09_21-45-12.png
AnyDeskMSI_2022-05-09_21-45-12.png
  425   Sun May 8 17:43:57 2022 NHAIDA FEE64 Layout
AIDA FEE64 layout with respect to the beam.
Attachment 1: AIDA_Adapter_Configuration(1).png
AIDA_Adapter_Configuration(1).png
  424   Sun May 8 15:10:45 2022 TDSunday 8 May
DSSSD bias and FEE64 power OFF during Ge install



Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.

Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists

Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass) 

The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads.
Attachment 1: 20220508_160103.jpg
20220508_160103.jpg
Attachment 2: 20220508_160115.jpg
20220508_160115.jpg
Attachment 3: 20220508_160128.jpg
20220508_160128.jpg
Attachment 4: 20220508_160140.jpg
20220508_160140.jpg
Attachment 5: 20220508_160720.jpg
20220508_160720.jpg
Attachment 6: 20220508_160804.jpg
20220508_160804.jpg
  423   Sat May 7 13:27:39 2022 TDAIDA configuration
Adaptor PCB link settings

n+n FEE64s

aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd

p+n FEE64s

   top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd

   top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd

AIDA PSU

   top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16


ASIC settings 2021Apr29-13-16-00
     slow comparator 0x64 -> 0xa

BNC PB-5 settings
    amplitude 1.000V
    attenuation x1
    tail pulse
    tau_d 1ms
    polarity +
    frequency 25Hz
  422   Sat May 7 09:55:53 2022 TDSaturday 7 May
Attachments 1 & 2 per FEE64 1.8.L spectra 
 pulser peak widths aida01 121 ch FWHM, aida02 517 ch FWHM

Attachments 3 & 4 per FEE64 1.8.W spectra 20us FSR

Attachment 5 per FEE64 rate spectra

Attachment 6 - grafana DSSSD bias/leakage current for previous 7 days showing effect of day/night temp variations

Attachments 7 & 8 - merger & tape server c. 14Mb/s to disk - no errors reported by servers

Attachments 9-12 - system wide checks - all OK

Attachment 13 ADC data item stats

Attachment 14 FEE64 temps OK

Attachment 15 DSSSD bias & leakage currents OK

16.15 aida05 zero ADC data items stats
      merger restart, DAQ STOP/Go cycle to recover


      activity is S4 area near AIDA DSSSDs as install of DEGAS detectors commences

19.50 DEGAS install continues

      DSSSD bias and FEE64 power OFF
 
Attachment 1: Screenshot_from_2022-05-07_10-53-56.png
Screenshot_from_2022-05-07_10-53-56.png
Attachment 2: Screenshot_from_2022-05-07_10-52-54.png
Screenshot_from_2022-05-07_10-52-54.png
Attachment 3: Screenshot_from_2022-05-07_10-51-33.png
Screenshot_from_2022-05-07_10-51-33.png
Attachment 4: Screenshot_from_2022-05-07_10-51-06.png
Screenshot_from_2022-05-07_10-51-06.png
Attachment 5: Screenshot_from_2022-05-07_10-47-13.png
Screenshot_from_2022-05-07_10-47-13.png
Attachment 6: Screenshot_from_2022-05-07_10-46-40.png
Screenshot_from_2022-05-07_10-46-40.png
Attachment 7: Screenshot_from_2022-05-07_10-46-26.png
Screenshot_from_2022-05-07_10-46-26.png
Attachment 8: Screenshot_from_2022-05-07_10-46-16.png
Screenshot_from_2022-05-07_10-46-16.png
Attachment 9: Screenshot_from_2022-05-07_10-45-34.png
Screenshot_from_2022-05-07_10-45-34.png
Attachment 10: Screenshot_from_2022-05-07_10-45-28.png
Screenshot_from_2022-05-07_10-45-28.png
Attachment 11: Screenshot_from_2022-05-07_10-45-23.png
Screenshot_from_2022-05-07_10-45-23.png
Attachment 12: Screenshot_from_2022-05-07_10-45-17.png
Screenshot_from_2022-05-07_10-45-17.png
Attachment 13: Screenshot_from_2022-05-07_10-45-08.png
Screenshot_from_2022-05-07_10-45-08.png
Attachment 14: Screenshot_from_2022-05-07_10-43-29.png
Screenshot_from_2022-05-07_10-43-29.png
Attachment 15: Screenshot_from_2022-05-07_10-43-14.png
Screenshot_from_2022-05-07_10-43-14.png
  421   Fri May 6 09:31:41 2022 TD NHFriday 6 May
10:30 DAQ appears to have stopped

      Attachment 2 - DSSSD bias and leakage current OK

      Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero

      RESET/SETUP/GO aida08 *only*
      DAQ STOP *all* FEE64s


      Attachment 6 - FEE64 temperatures OK

      Attachments 7-23 - system wide checks - all OK

10:48 DAQ GO *all* FEE64s
      all spectra zero'd

      attachment 29 - ADC data items
       n+n FEE64s aida02, 4, 6 & 8 c. 250k
       p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k

      attachments 26 & 27 1.8.L spectra
       aida01 peak width 112ch FWHM
       aida02 peak width 485 ch FWHM

      attachments 24 & 25 1.8.W spectra 20us FSR
      
      attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data

      attachment 31 - aida05 also producing disc data

      attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently? 


18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
      Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1

      Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday.
Attachment 1: Screenshot_from_2022-05-06_10-27-27.png
Screenshot_from_2022-05-06_10-27-27.png
Attachment 2: Screenshot_from_2022-05-06_10-26-59.png
Screenshot_from_2022-05-06_10-26-59.png
Attachment 3: Screenshot_from_2022-05-06_10-26-33.png
Screenshot_from_2022-05-06_10-26-33.png
Attachment 4: Screenshot_from_2022-05-06_10-25-44.png
Screenshot_from_2022-05-06_10-25-44.png
Attachment 5: Screenshot_from_2022-05-06_10-24-54.png
Screenshot_from_2022-05-06_10-24-54.png
Attachment 6: Screenshot_from_2022-05-06_10-42-18.png
Screenshot_from_2022-05-06_10-42-18.png
Attachment 7: Screenshot_from_2022-05-06_10-41-06.png
Screenshot_from_2022-05-06_10-41-06.png
Attachment 8: Screenshot_from_2022-05-06_10-41-00.png
Screenshot_from_2022-05-06_10-41-00.png
Attachment 9: Screenshot_from_2022-05-06_10-40-54.png
Screenshot_from_2022-05-06_10-40-54.png
Attachment 10: Screenshot_from_2022-05-06_10-40-09.png
Screenshot_from_2022-05-06_10-40-09.png
Attachment 11: Screenshot_from_2022-05-06_10-39-07.png
Screenshot_from_2022-05-06_10-39-07.png
Attachment 12: Screenshot_from_2022-05-06_10-38-58.png
Screenshot_from_2022-05-06_10-38-58.png
Attachment 13: Screenshot_from_2022-05-06_10-38-35.png
Screenshot_from_2022-05-06_10-38-35.png
Attachment 14: Screenshot_from_2022-05-06_10-38-21.png
Screenshot_from_2022-05-06_10-38-21.png
Attachment 15: Screenshot_from_2022-05-06_10-38-16.png
Screenshot_from_2022-05-06_10-38-16.png
Attachment 16: Screenshot_from_2022-05-06_10-38-10.png
Screenshot_from_2022-05-06_10-38-10.png
Attachment 17: Screenshot_from_2022-05-06_10-38-02.png
Screenshot_from_2022-05-06_10-38-02.png
Attachment 18: Screenshot_from_2022-05-06_10-37-52.png
Screenshot_from_2022-05-06_10-37-52.png
Attachment 19: Screenshot_from_2022-05-06_10-37-46.png
Screenshot_from_2022-05-06_10-37-46.png
Attachment 20: Screenshot_from_2022-05-06_10-37-39.png
Screenshot_from_2022-05-06_10-37-39.png
Attachment 21: Screenshot_from_2022-05-06_10-37-31.png
Screenshot_from_2022-05-06_10-37-31.png
Attachment 22: Screenshot_from_2022-05-06_10-36-24.png
Screenshot_from_2022-05-06_10-36-24.png
Attachment 23: Screenshot_from_2022-05-06_10-35-52.png
Screenshot_from_2022-05-06_10-35-52.png
Attachment 24: Screenshot_from_2022-05-06_10-56-31.png
Screenshot_from_2022-05-06_10-56-31.png
Attachment 25: Screenshot_from_2022-05-06_10-55-56.png
Screenshot_from_2022-05-06_10-55-56.png
Attachment 26: Screenshot_from_2022-05-06_10-55-13.png
Screenshot_from_2022-05-06_10-55-13.png
Attachment 27: Screenshot_from_2022-05-06_10-53-53.png
Screenshot_from_2022-05-06_10-53-53.png
Attachment 28: Screenshot_from_2022-05-06_10-51-00.png
Screenshot_from_2022-05-06_10-51-00.png
Attachment 29: Screenshot_from_2022-05-06_10-50-25.png
Screenshot_from_2022-05-06_10-50-25.png
Attachment 30: Screenshot_from_2022-05-06_11-09-15.png
Screenshot_from_2022-05-06_11-09-15.png
Attachment 31: Screenshot_from_2022-05-06_11-02-33.png
Screenshot_from_2022-05-06_11-02-33.png
Attachment 32: Screenshot_from_2022-05-06_11-19-48.png
Screenshot_from_2022-05-06_11-19-48.png
Attachment 33: Screenshot_from_2022-05-06_11-19-27.png
Screenshot_from_2022-05-06_11-19-27.png
  420   Wed May 4 16:50:04 2022 NHAIDA 4th May
bPlas has been mostly cabled in now

Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.

CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too

Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty

ASIC check-loads didn't fix
Try powercycle... also didn't fix

Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.

Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast
Attachment 1: Screenshot_20220504_174929.png
Screenshot_20220504_174929.png
Attachment 2: Screenshot_20220504_175201.png
Screenshot_20220504_175201.png
  419   Tue May 3 14:15:17 2022 NHAIDA Updates 2 May
PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16

These FEE64s also have the highest rates of discriminator data?

> The snout endcap has been sealed with new copper tape over the red tape
> 
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
> 
> 
> 
> -
> 
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
> 
> -
> 
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
> 
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
> 
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May  2 14:04:03 2022
> Duration   : 324823 seconds
> --------------------------------------
> MBS Events   (M):     912899 (2 Hz)
> AIDA Words   (W): 7472078760 (-3441 Hz)
> ADC Words    (A): 2742871865 (-4778 Hz)
> -- Decays    (D): 2728717223 (-4821 Hz)
> -- Implants  (I):   14154642 (43 Hz)
> Info Words   (F): 4729206895 (1 kHz)
> -- SYNC48    (S):    2093328 (6 Hz)
> -- SYNC63    (s):    2093328 (6 Hz)
> -- PAUSE     (P):    3823161 (11 Hz)
> -- RESUME    (R):    3823161 (11 Hz)
> -- Discrim   (D): 1390520565 (4 kHz)
> -- Scaler    (C): 3326853352 (3 kHz)
> -- Unknown   (U):          0 (0 Hz)
> Timewarps    (T):         20 (0 Hz)
> Dead Time    (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
>  FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
>    1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
>    2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
>    3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
>    4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
>    5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
>    6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
>    7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
>    8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
>    9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
>   10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
>   11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
>   12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
>   13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
>   14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
>   15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
>   16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9
  418   Mon May 2 13:08:49 2022 NHAIDA Updates 2 May
The snout endcap has been sealed with new copper tape over the red tape

Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive



-

the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s

-

Preliminary alpha background analysis (over weekend, 90 hours statistics)

Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Apr 28 19:50:20 2022
Last event : Mon May  2 14:04:03 2022
Duration   : 324823 seconds
--------------------------------------
MBS Events   (M):     912899 (2 Hz)
AIDA Words   (W): 7472078760 (-3441 Hz)
ADC Words    (A): 2742871865 (-4778 Hz)
-- Decays    (D): 2728717223 (-4821 Hz)
-- Implants  (I):   14154642 (43 Hz)
Info Words   (F): 4729206895 (1 kHz)
-- SYNC48    (S):    2093328 (6 Hz)
-- SYNC63    (s):    2093328 (6 Hz)
-- PAUSE     (P):    3823161 (11 Hz)
-- RESUME    (R):    3823161 (11 Hz)
-- Discrim   (D): 1390520565 (4 kHz)
-- Scaler    (C): 3326853352 (3 kHz)
-- Unknown   (U):          0 (0 Hz)
Timewarps    (T):         20 (0 Hz)
Dead Time    (X): 5024.43 s (1.54682%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
   1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
   2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
   3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
   4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
   5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
   6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
   7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
   8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
   9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
  10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
  11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
  12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
  13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
  14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
  15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
  16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9
Attachment 1: elog_temps1.png
elog_temps1.png
Attachment 2: alpha_1d.png
alpha_1d.png
  417   Mon May 2 09:52:20 2022 OHShifter checklist 2022
Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 3)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- IGNORE FOR TIME BEING Options database monitor (Top middle terminal screen 2, workspace 3)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 5)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 3)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 3)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 3)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 3)

- Perform the five system wide checks (Web browser tab screen 1, workspace 3)
     - Ignore the system wide checks for ADC calibration (ADCs are turned off so they will all fail)
     - Check memory information also fails.
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 5)
  416   Thu Apr 28 18:45:09 2022 NHAlpha Background Run
Runs are in /TapeData/BGAPR22 located on 8TB disk "SecondDrive" (N.B. 2A experiment is using ThirdDrive)

Old runs and test files were removed from disk. Left is an unknown S481 folder (fibre implant test?) and Patrick
chown to npg:npg as uid changed from old SL6 to CentOS 7

Bias DSSDs, leakage current higher than before (maybe the sheet isn't covering the snout)
Pulser set to OFF
ASIC Thresholds 0x64, rates mostly 0 (good)
Edge strips on aida02 and aida06 show non zero, and lots in aida04 asic 1 are high, will recheck connector tomorrow in hope

Connecting to merger and rate is 60k items/sec with random jumps to 400k items/sec
Start xfer to tape server: Rate approx 100 KB/s (360 MB per hour, 2.88 GB per 8 hours)

19:51 - Will leave overnight running to collect alpha

Fri 29 Apr

14:38 - Run paused for MBS tests with FRS

16:50 - Resume alpha collection over weekend
Pulser OFF, 0x64 thresholds
Attachment 1: AnyDeskMSI_2022-04-28_19-47-33.png
AnyDeskMSI_2022-04-28_19-47-33.png
Attachment 2: AnyDeskMSI_2022-04-28_19-47-56.png
AnyDeskMSI_2022-04-28_19-47-56.png
Attachment 3: AnyDeskMSI_2022-04-28_19-48-13.png
AnyDeskMSI_2022-04-28_19-48-13.png
  415   Wed Apr 27 15:47:46 2022 NH, HA, SA, CJAIDA 27 Apr 22
Remount snout in S4 (photos to come) 
Alignment with DEGAS OK but a bit short...  DEGAS platform will be brought 31 mm closer to AIDA by rail adjustment

Can mount bPlas booster boards on old ITEM frames and also partly on AIDA snout (isolated)

-

Reconnect all adapter boards and grounds, bias AIDA...

HV OK. We still have a light leak at the snout endcap. "Fixed" with sheet for now

HV leakage currents figure 1

Start AIDA DAQ and check rates.. all seem OK. aida07 is very noisy other p+n good
Notice some issues in aida04, check and ribbon cable slightly loose on one side of adapter
No change to other FEEs.
Aida07 was inspected but nothing obviously wrong

Rates figure 2
Rate spectra figure 3
Wavesform figures 4 (p+n), 5 (n+n)
Pulser peak withs
 p+n figure 6, width ~  70 channels (50 keV)
 n+n figure 7, width ~ 570 channels (400 keV)

n.b. double peak in aida14 is an artifact from an uncleared histogram

DSSDs debiased to allow work on bPlas cabling without tripping DSSDs
Attachment 1: 2022-04-27_16-42-37_HV.png
2022-04-27_16-42-37_HV.png
Attachment 2: 022-04-27_16-39-58_Rates.png
022-04-27_16-39-58_Rates.png
Attachment 3: 2022-04-27_16-40-23_RatesSpectra.png
2022-04-27_16-40-23_RatesSpectra.png
Attachment 4: 2022-04-27_16-41-46_pnwaves.png
2022-04-27_16-41-46_pnwaves.png
Attachment 5: 2022-04-27_16-42-20_nn_waves.png
2022-04-27_16-42-20_nn_waves.png
Attachment 6: 2022-04-27_16-43-30_pn_pulser_70ch.png
2022-04-27_16-43-30_pn_pulser_70ch.png
Attachment 7: 2022-04-27_16-44-18._nn_pulser_568ch.png
2022-04-27_16-44-18._nn_pulser_568ch.png
  414   Fri Apr 8 08:59:42 2022 OH, NHFriday 8th April
09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 
Attachment 1: 220408_1000_bias.png
220408_1000_bias.png
Attachment 2: 220409_1010_Stats.png
220409_1010_Stats.png
Attachment 3: 220409_1011_rate.png
220409_1011_rate.png
Attachment 4: 220408_1358_bias.png
220408_1358_bias.png
Attachment 5: 220408_1444_rate.png
220408_1444_rate.png
Attachment 6: 220408_1444_stats.png
220408_1444_stats.png
Attachment 7: 220408_1449_layout7.png
220408_1449_layout7.png
Attachment 8: 220408_1451_layout8.png
220408_1451_layout8.png
Attachment 9: 220308_1459_bias.png
220308_1459_bias.png
Attachment 10: 220408_1503_stats_post_light.png
220408_1503_stats_post_light.png
Attachment 11: 2022-04-08_16-26-30_Stats_No_Pulser.png
2022-04-08_16-26-30_Stats_No_Pulser.png
  413   Thu Apr 7 12:42:14 2022 OH, NH, TDThursday 7 April
Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409

Attachments 1 & 2 1.8.L spectra
  p+n FEE64 aida10 pulser peak width 97 ch FWHM
  n+n FEE64 aida04 pulser peak width 550 ch FWHM

Attachment 3 rate spectra
  missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?

Attachments 4 & 5 1.8.W spectra 

Attachment 6 ADC data item rates
 aida10 OK, all others c. 300-400k

Attachment 7 FEE64 temperatures OK

Attachment 8 DSSSD bias & leakage currents OK
Attachment 1: Screenshot_from_2022-04-07_13-40-55.png
Screenshot_from_2022-04-07_13-40-55.png
Attachment 2: Screenshot_from_2022-04-07_13-40-02.png
Screenshot_from_2022-04-07_13-40-02.png
Attachment 3: Screenshot_from_2022-04-07_13-37-17.png
Screenshot_from_2022-04-07_13-37-17.png
Attachment 4: Screenshot_from_2022-04-07_13-35-43.png
Screenshot_from_2022-04-07_13-35-43.png
Attachment 5: Screenshot_from_2022-04-07_13-35-23.png
Screenshot_from_2022-04-07_13-35-23.png
Attachment 6: Screenshot_from_2022-04-07_13-34-34.png
Screenshot_from_2022-04-07_13-34-34.png
Attachment 7: Screenshot_from_2022-04-07_13-34-12.png
Screenshot_from_2022-04-07_13-34-12.png
Attachment 8: Screenshot_from_2022-04-07_13-33-53.png
Screenshot_from_2022-04-07_13-33-53.png
  412   Wed Apr 6 10:09:21 2022 OH, NH, TDWednesday 6 April
DSSSD bias OFF. DAQ STOPped

Attachment 6 FEE64 temps OK

Attachment 1-5 system wide checks OK 
Attachment 1: Screenshot_from_2022-04-06_11-12-59.png
Screenshot_from_2022-04-06_11-12-59.png
Attachment 2: Screenshot_from_2022-04-06_11-12-48.png
Screenshot_from_2022-04-06_11-12-48.png
Attachment 3: Screenshot_from_2022-04-06_11-12-38.png
Screenshot_from_2022-04-06_11-12-38.png
Attachment 4: Screenshot_from_2022-04-06_11-10-06.png
Screenshot_from_2022-04-06_11-10-06.png
Attachment 5: Screenshot_from_2022-04-06_11-09-57.png
Screenshot_from_2022-04-06_11-09-57.png
Attachment 6: Screenshot_from_2022-04-06_11-09-29.png
Screenshot_from_2022-04-06_11-09-29.png
  411   Tue Apr 5 12:46:52 2022 TDMSL type BB18(DS)-1000 triple - cable lengths
OH/NH disassembling current AIDA triple snout and DSSSD stack.

Length of ribbon cables (including connector) outside snout as follows

p+n junction side 9cm & 15cm
n+n ohmic side 10.5cm & 16cm 


Update
p+n junction side: 6.5cm, 10.5 cm (used  7.5, 11.5 for 1cm slack)
n+n ohmic side:  11.0 cm, 15.0 cm (used 12.0, 16.0 for 1cm slack)
Measurement from SAMTEC connector to exit from the rubber seals in AIDA snout.
  410   Tue Apr 5 08:52:16 2022 TDTuesday 5 April
DSSSD bias OFF. DAQ STOPped

Attachment 1 FEE64 temps OK

Attachment 2 system wide checks OK *except* WR decoder (attachment 2) and FPGA ts errors no longer works

Attachment 6 System wide check - 'Synchronise ASIC clocks'

Attachment 5 System wide check - check ADC calibration - all 16x FEE64s fail

Attachment 4 System wide check - WR decoder status - aida09 WR status 0x10 clears

Repeat Synchronsie ASIC clocks

Attachment 3 System wide check - WR decoder status - aida05 WR status 0x10 clears
             
Attachment 1: Screenshot_from_2022-04-05_09-52-42.png
Screenshot_from_2022-04-05_09-52-42.png
Attachment 2: Screenshot_from_2022-04-05_10-05-59.png
Screenshot_from_2022-04-05_10-05-59.png
Attachment 3: Screenshot_from_2022-04-05_16-10-31.png
Screenshot_from_2022-04-05_16-10-31.png
Attachment 4: Screenshot_from_2022-04-05_16-09-38.png
Screenshot_from_2022-04-05_16-09-38.png
Attachment 5: Screenshot_from_2022-04-05_16-09-30.png
Screenshot_from_2022-04-05_16-09-30.png
Attachment 6: Screenshot_from_2022-04-05_16-08-44.png
Screenshot_from_2022-04-05_16-08-44.png
  409   Mon Apr 4 09:19:49 2022 TDMonday 4 April
AIDA adaptor PCB config - all LKs removed *except*

aida04 & aida08 LK1

aida02 & aida07 LK2, LK3 & LK4

aida09, aida01, aida10, aida11, aida15, aida12 & aida16 LK2 & LK4



10.05 check all ASICs all FEE64s
      zero histograms

Attachment 1 per FEE64 rate spectra

Attachment 2 per FEE64 p+n 1.8.L spectra - aida10 peak width 79 ch FWHM

Attachment 3 per FEE64 p+n 1.8.W spectra 20us FSR

Attachment 4 per FEE64 n+n 1.8.W spectra 20us FSR

Attachments 5-8 - system wide checks - all OK *except*

  FPGA timestamp errors no longer works
  aida05 & aida09 report WR decoder error 0x10

Attachment 9 ADC data item stats

Attachment 10 FEE64 temps OK

Attachment 11 DSSSD bias & leakage current OK

Attachment 12 per FEE64 n+n 1.8.L spectra - aida08 peak width 369 ch FWHM


14.25 DAQ STOP
      DSSSD bias OFF

      AIDA triple snout removed from AIDA stand and moved to table by aida-gsi (Messeheute)
Attachment 1: Screenshot_from_2022-04-04_10-18-36.png
Screenshot_from_2022-04-04_10-18-36.png
Attachment 2: Screenshot_from_2022-04-04_10-17-30.png
Screenshot_from_2022-04-04_10-17-30.png
Attachment 3: Screenshot_from_2022-04-04_10-14-53.png
Screenshot_from_2022-04-04_10-14-53.png
Attachment 4: Screenshot_from_2022-04-04_10-14-24.png
Screenshot_from_2022-04-04_10-14-24.png
Attachment 5: Screenshot_from_2022-04-04_10-13-09.png
Screenshot_from_2022-04-04_10-13-09.png
Attachment 6: Screenshot_from_2022-04-04_10-12-41.png
Screenshot_from_2022-04-04_10-12-41.png
Attachment 7: Screenshot_from_2022-04-04_10-12-33.png
Screenshot_from_2022-04-04_10-12-33.png
Attachment 8: Screenshot_from_2022-04-04_10-12-26.png
Screenshot_from_2022-04-04_10-12-26.png
Attachment 9: Screenshot_from_2022-04-04_10-11-09.png
Screenshot_from_2022-04-04_10-11-09.png
Attachment 10: Screenshot_from_2022-04-04_10-10-58.png
Screenshot_from_2022-04-04_10-10-58.png
Attachment 11: Screenshot_from_2022-04-04_10-09-37.png
Screenshot_from_2022-04-04_10-09-37.png
Attachment 12: Screenshot_from_2022-04-04_10-29-08.png
Screenshot_from_2022-04-04_10-29-08.png
  408   Sun Apr 3 10:56:11 2022 TDSunday 3 April
all system wide checks OK *except* aida10 failed ADC calibration

Attachment 1 ADC data item stats
Attachment 2 FEE64 temps OK
Attachment 3 DSSSD bias & leakage currents OK





DSSSD bias to left (looking upstream) wafer *only*
      FEE64s 10 & 12, 14 & 16

Attachments 4 & 5 1.8.W spectra 20us FSR

Attachment 6 ADC data item stats

Attachment 7 DSSSD bias & leakage current




ASIC check all FEE64s

DSSSD bias to middle (looking upstream) wafer *only*
      FEE64s 1 & 3, 5 & 7

Attachments 8 & 9 1.8.W spectra 20us FSR

Attachment 10 ADC data item stats

Attachment 11 DSSSD bias & leakage current





ASIC check all FEE64s

DSSSD bias to right (looking upstream) wafer *only*
      FEE64s 9 & 11, 13 & 15

Attachments 12 & 13 1.8.W spectra 20us FSR

Attachment 14 ADC data item stats

Attachment 15 DSSSD bias & leakage current



Conclusions

ADC data item stats -> 300-400k

n+n 1.8.W spectra show large signal excursions

p+n 1.8.W spectra for biased Si wafers OK - unbiased Si wafers show large signal excursions


DSSSD#1 leakage currents 1.850+2.220+1.990=6.06uA    cf. 5.095uA with all 3 wafers biased 
DSSSD#2 leakage currents 1.510+1.580+2.350=5.44uA    cf. 5.235uA 


Right wafer (looking upstream)
             
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.135 -1.475 -1.645 -1.750 -1.805
DSSSD#2 I_l(uA) -1.625 -2.020 -2.205 -2.320 -2.360

Middle wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.540 -1.870 -2.035 -2.150 -2.210
DSSSD#2 I_l(uA) -0.985 -1.285 -1.465 -1.575 -1.615

Left wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -0.735 -0.915 -1.005 -1.080 -1.110
DSSSD#2 I_l(uA) -0.865 -1.205 -1.400 -1.510 -1.560

Bias versus leakage cuurent characteristic of each wafer of each DSSSD indicates diode behaviour.

Note LK1 not fitted - presumably LK3 of mid/top p+n FEE64 adaptor PCBs (aida01 and aida05) connecting n+n bias to gnd is sufficient to establish circuit.




15.50 Re-connect bias to all 3x Si wafers p+n junction side bias rings

ASIC check all FEE64s

Attachments 16 & 17 1.8.W spectra 20us FSR

Attachment 18 ADC data item stats

Attachment 19 DSSSD bias & leakage current

Conclusion - status quo ante restored



16.40 Remove aida01 & aida05 LK3, add aida04 & aida08 LK1

ASIC check all FEE64s

Attachments 20 & 21 1.8.W spectra 20us FSR

Attachment 22 ADC data item stats

Attachment 23 DSSSD bias & leakage current

Conclusion - no change



17.00 Add Thorn-Lab HV filters DSSSD#1 & #2


ASIC check all FEE64s

Attachments 24 & 25 1.8.W spectra 20us FSR

Attachment 26 ADC data item stats

Attachment 27 DSSSD bias & leakage current

Conclusion - aida09-aida12 no change, aida13-aida16 -> 300-400k
             I do not know why there's a difference between DSSSD#1 & #2 but more important point (probably) no improvement aida09-aida12
             Thorn Lab HV filters removed - status quo ante restored


18.02 Remove LK2 & LK4 all top p+n adaptor PCBs (aida13, 5, 14, 9, 1, & 10)

ASIC check all FEE64s

Attachments 28 & 29 1.8.W spectra 20us FSR

Attachment 30 ADC data item stats

Attachment 31 DSSSD bias & leakage current

Conclusion - no change cf. 16.40 test



19.10 Re-order power cabling at AIDA FEE64 PSUs from ...

Top    1-3 2-4 9-10 11-12
Bottom 5-7 6-8 13-14 15-16

to ...

Top    1-2 3-4 9-10 11-12
Bottom 5-6 7-8 13-14 15-16


ASIC check all FEE64s

Attachments 32 & 33 1.8.W spectra 20us FSR

Attachment 34 ADC data item stats

Attachment 35 DSSSD bias & leakage current

Conclusion - all p+n FEE64 rates c. 100k, or less, n+n FEE64 rates c. 400k

FEE64 adaptor PCB config

LK2-LK3-LK4 fitted for all bottom p+n FEE64s (aida11, 3, 12, 15, 7, 16)

LK1 fitted n+n FEE64s aida06 & aida02 *only*
Attachment 1: Screenshot_from_2022-04-03_11-54-21.png
Screenshot_from_2022-04-03_11-54-21.png
Attachment 2: Screenshot_from_2022-04-03_11-54-04.png
Screenshot_from_2022-04-03_11-54-04.png
Attachment 3: Screenshot_from_2022-04-03_11-53-15.png
Screenshot_from_2022-04-03_11-53-15.png
Attachment 4: Screenshot_from_2022-04-03_13-07-21.png
Screenshot_from_2022-04-03_13-07-21.png
Attachment 5: Screenshot_from_2022-04-03_13-06-48.png
Screenshot_from_2022-04-03_13-06-48.png
Attachment 6: Screenshot_from_2022-04-03_13-06-34.png
Screenshot_from_2022-04-03_13-06-34.png
Attachment 7: Screenshot_from_2022-04-03_13-05-24.png
Screenshot_from_2022-04-03_13-05-24.png
Attachment 8: Screenshot_from_2022-04-03_13-21-31.png
Screenshot_from_2022-04-03_13-21-31.png
Attachment 9: Screenshot_from_2022-04-03_13-21-07.png
Screenshot_from_2022-04-03_13-21-07.png
Attachment 10: Screenshot_from_2022-04-03_13-20-43.png
Screenshot_from_2022-04-03_13-20-43.png
Attachment 11: Screenshot_from_2022-04-03_13-20-25.png
Screenshot_from_2022-04-03_13-20-25.png
Attachment 12: Screenshot_from_2022-04-03_13-32-24.png
Screenshot_from_2022-04-03_13-32-24.png
Attachment 13: Screenshot_from_2022-04-03_13-31-50.png
Screenshot_from_2022-04-03_13-31-50.png
Attachment 14: Screenshot_from_2022-04-03_13-31-32.png
Screenshot_from_2022-04-03_13-31-32.png
Attachment 15: Screenshot_from_2022-04-03_13-31-06.png
Screenshot_from_2022-04-03_13-31-06.png
Attachment 16: Screenshot_from_2022-04-03_15-52-42.png
Screenshot_from_2022-04-03_15-52-42.png
Attachment 17: Screenshot_from_2022-04-03_15-51-49.png
Screenshot_from_2022-04-03_15-51-49.png
Attachment 18: Screenshot_from_2022-04-03_15-51-34.png
Screenshot_from_2022-04-03_15-51-34.png
Attachment 19: Screenshot_from_2022-04-03_15-50-53.png
Screenshot_from_2022-04-03_15-50-53.png
Attachment 20: Screenshot_from_2022-04-03_16-40-15.png
Screenshot_from_2022-04-03_16-40-15.png
Attachment 21: Screenshot_from_2022-04-03_16-39-43.png
Screenshot_from_2022-04-03_16-39-43.png
Attachment 22: Screenshot_from_2022-04-03_16-39-34.png
Screenshot_from_2022-04-03_16-39-34.png
Attachment 23: Screenshot_from_2022-04-03_16-39-22.png
Screenshot_from_2022-04-03_16-39-22.png
Attachment 24: Screenshot_from_2022-04-03_16-59-59.png
Screenshot_from_2022-04-03_16-59-59.png
Attachment 25: Screenshot_from_2022-04-03_16-59-33.png
Screenshot_from_2022-04-03_16-59-33.png
Attachment 26: Screenshot_from_2022-04-03_16-59-10.png
Screenshot_from_2022-04-03_16-59-10.png
Attachment 27: Screenshot_from_2022-04-03_16-59-07.png
Screenshot_from_2022-04-03_16-59-07.png
Attachment 28: Screenshot_from_2022-04-03_18-01-39.png
Screenshot_from_2022-04-03_18-01-39.png
Attachment 29: Screenshot_from_2022-04-03_18-00-56.png
Screenshot_from_2022-04-03_18-00-56.png
Attachment 30: Screenshot_from_2022-04-03_18-00-43.png
Screenshot_from_2022-04-03_18-00-43.png
Attachment 31: Screenshot_from_2022-04-03_18-00-40.png
Screenshot_from_2022-04-03_18-00-40.png
Attachment 32: Screenshot_from_2022-04-03_19-10-37.png
Screenshot_from_2022-04-03_19-10-37.png
Attachment 33: Screenshot_from_2022-04-03_19-10-11.png
Screenshot_from_2022-04-03_19-10-11.png
Attachment 34: Screenshot_from_2022-04-03_19-10-00.png
Screenshot_from_2022-04-03_19-10-00.png
Attachment 35: Screenshot_from_2022-04-03_19-09-36.png
Screenshot_from_2022-04-03_19-09-36.png
  407   Sat Apr 2 10:42:49 2022 TDSaturday 2 April
11.10 Found aida08 not running - merger/TapeServer stalled

Restart required power cycle of all FEE64s
*all* ADC calibrations failed during system wide checks it was necessary to use FADC align &amp; control (act on all FEE64 modules selected) to calibrate - first pass 
12/16 calibrated, successive individual re-calibrations calibrated remaining FEE64s

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x2f

BNC PB-5
amplitude 1.0V
attenuation x1
polarity +
tail pulse
tau_d 1ms
rate 25Hz

Attachments 1-4 1.8.W n+n spectra 20us, 200us, 2ms, 20ms FSR
Attachments 5-8 1.8.W p+n spectra 20ms, 2ms, 200us, 20us FSR

Attachment 9 per FEE64 1.8.L spectra n+n strips
 aida04 pulser peak width 387 ch FWHM
Attachment 10 per FEE64 1.8.L spectra p+n strips
 aida10 pulser peak width 93 ch FWHM

Attachment 11 per FEE64 Rate spectra

Attachments 12-15 all system wide checks OK
Attachment 16 ADC data item stats
Attachment 17 FEE64 temps OK
Attachment 18 DSSSD bias & leakage currents OK

15.45 check of ASIC settings cf. https://www2.ph.ed.ac.uk/~td/AIDA/Information/Default%20AIDA%20ASIC%20Parameters.txt

Differences from default AIDA ASIC parameters
--------------------------------------------- 

                                            Input Polarity
Offset Title                                Positive   Negative            Notes

12     slow comparator threshold            0x2f->0xa                     100keV nominal
 
14     fast comparator threshold HEC        0xf->0x02                     200MeV nominal
15     fast comparator threshold LEC/MEC    0xf->0xff                     > LEC/MEC nominal FSR

18     preAmp reference                     0xb2       0x20->0x30

21     Ibias LF feedback                    0x08       0xb->0xf

26     diode link threshold                 0xca->0xbf 0x11->0x23         decrease switch threshold

(front) p+n junction strips - positive input polarity
(back) n+n ohmic strips - negative input polarity

All looks OK - variations from default settings are those used at RIKEN

16:45 FEE64 adaptor PCB config

    LK1 LK2 LK3 LK4
p+n n/c o   *   o
n+n o   o   n/c o

o   = jumper fitted
n/c = jumper not fitted

*   = top, middle FEE64s - n+n bias ring ground for Si wafer #2
      bottom, middle FEE64s - DSSSD PCB gnd

DSSSD#1 & #2 bias via standard coaxial cable to top p+n FEE64 adaptor PCBs

17.05 aida09 or aida13 cable braid not secured to ground - cable tag dia too large - now sandwiched between correctly sized cable tags 
      no obvious difference to adc data item rates


17.20 LK1 removed for all n+n PCBs
      attachment 19 - adc data item stats

18.55 LK2/LK4 removed for all n+n PCBs
      attachment 20 - adc data item stats

19.10 power cycle FEE64s
      attachment 21 - adc data item stats
Attachment 1: Screenshot_from_2022-04-02_11-50-09.png
Screenshot_from_2022-04-02_11-50-09.png
Attachment 2: Screenshot_from_2022-04-02_11-49-44.png
Screenshot_from_2022-04-02_11-49-44.png
Attachment 3: Screenshot_from_2022-04-02_11-49-20.png
Screenshot_from_2022-04-02_11-49-20.png
Attachment 4: Screenshot_from_2022-04-02_11-49-02.png
Screenshot_from_2022-04-02_11-49-02.png
Attachment 5: Screenshot_from_2022-04-02_11-48-38.png
Screenshot_from_2022-04-02_11-48-38.png
Attachment 6: Screenshot_from_2022-04-02_11-47-54.png
Screenshot_from_2022-04-02_11-47-54.png
Attachment 7: Screenshot_from_2022-04-02_11-46-51.png
Screenshot_from_2022-04-02_11-46-51.png
Attachment 8: Screenshot_from_2022-04-02_11-46-11.png
Screenshot_from_2022-04-02_11-46-11.png
Attachment 9: Screenshot_from_2022-04-02_11-44-14.png
Screenshot_from_2022-04-02_11-44-14.png
Attachment 10: Screenshot_from_2022-04-02_11-41-09.png
Screenshot_from_2022-04-02_11-41-09.png
Attachment 11: Screenshot_from_2022-04-02_11-39-57.png
Screenshot_from_2022-04-02_11-39-57.png
Attachment 12: Screenshot_from_2022-04-02_11-39-02.png
Screenshot_from_2022-04-02_11-39-02.png
Attachment 13: Screenshot_from_2022-04-02_11-38-55.png
Screenshot_from_2022-04-02_11-38-55.png
Attachment 14: Screenshot_from_2022-04-02_11-38-47.png
Screenshot_from_2022-04-02_11-38-47.png
Attachment 15: Screenshot_from_2022-04-02_11-38-35.png
Screenshot_from_2022-04-02_11-38-35.png
Attachment 16: Screenshot_from_2022-04-02_11-38-19.png
Screenshot_from_2022-04-02_11-38-19.png
Attachment 17: Screenshot_from_2022-04-02_11-37-51.png
Screenshot_from_2022-04-02_11-37-51.png
Attachment 18: Screenshot_from_2022-04-02_11-37-30.png
Screenshot_from_2022-04-02_11-37-30.png
Attachment 19: Screenshot_from_2022-04-02_17-20-12.png
Screenshot_from_2022-04-02_17-20-12.png
Attachment 20: Screenshot_from_2022-04-02_18-54-11.png
Screenshot_from_2022-04-02_18-54-11.png
Attachment 21: Screenshot_from_2022-04-02_19-07-42.png
Screenshot_from_2022-04-02_19-07-42.png
  406   Wed Mar 9 10:15:56 2022 NH TDWednesday 9 March - noise tests contd.
11.15 - S4 is currently closed no access

If access possible (should be) plan is

Re-attach ribbon cable braids
Re-attach DSSSD middle wafer rear g/r (LK3)
Re-attach DSSSD ground (LK2?)

See difference in rates, expectation is braid should restore most p+n rates

Once status quo ante achieved explore ASIC settings to identify preamp feedback and shaping time options to stabilise preamp output and optimise signal:noise.

aida12 Peak width =    84.73
aida02 Peak width =   526.68
Attachment 1 - Stats
Attachment 2 - Per FEE Rates
Attachments 3,4 - Waveforms
Attachments 5,6 - Pulser Peaks
Attachment 7 - HV Bias. Leakage current recovered and does not seem to be ramping up

16.00 - Attach LK3 jumpers to aidas 1, 3, 5, 7 (DSSD ground and middle rear g/r)

Attachment 8 - Stats
Attachment 9 - Per FEE Rates
Attachment 10, 11 - Waveforms
Attachment 12, 13 - Pulser Peaks

aida12 Peak width =    78.51
aida02 Peak width =   480.90

No major change, FEES 1-8 are all noisy and FEES 9-16 are quiet
aida01, aida05 waveforms very noticably bad (100 kHz), aida03, aida07 waveforms seem OK


16.45 - Attach LK2, LK4 jumpers to all FEEs (field plane grounding)
No significant changes

Attachment 14 - Stats
Attachment 15 - Per FEE Rates
Attachment 16, 17 - Pulser Peaks
Attachment 18, 19 - Waveforms

aida12 Peak width =    71.28
aida02 Peak width =   458.18

Remove LK3 jumpers 

aida12 Peak width =    71.43 
aida02 Peak width =   496.05


Attachment 20 - Stats
Attachment 21 - Per FEE Rates
Attachment 22, 23 - Pulser Peaks
Attachment 24, 25 - Waveforms

Re-add jumpers - attachments 26-28

aida12 Peak width =    87.42
aida02 Peak width =   572.24

17.45 Directly bias from the p+n FEEs using SHV->BNC->Lemo - attachments 29-33

aida12 Peak width =    87.87
aida02 Peak width =   504.41

The correlation between high 'ADC data item' rate and the noise observed on the corresponding 1.8.W spectrum is not clear *except* for aida01 & aida05 - see also attachments 34-35

ADC data item stats - attachment 36

DSSSD bias & leakage current OK - attachment 37

FEE64 temperatures OK - attachment 38

all system wide checks OK *except aida11 clock fail status 6
Attachment 1: 2022-03-09_15-37-08_stats.png
2022-03-09_15-37-08_stats.png
Attachment 2: 2022-03-09_15-37-33_rates.png
2022-03-09_15-37-33_rates.png
Attachment 3: 2022-03-09_15-38-18_pn_waves.png
2022-03-09_15-38-18_pn_waves.png
Attachment 4: 022-03-09_15-38-59_nn_waves.png
022-03-09_15-38-59_nn_waves.png
Attachment 5: 2022-03-09_15-40-23_pn_widths.png
2022-03-09_15-40-23_pn_widths.png
Attachment 6: 2022-03-09_15-41-26_nn_widths.png
2022-03-09_15-41-26_nn_widths.png
Attachment 7: 2022-03-09_15-42-55_hv_bias.png
2022-03-09_15-42-55_hv_bias.png
Attachment 8: 2022-03-09_16-03-05_stats.png
2022-03-09_16-03-05_stats.png
Attachment 9: 2022-03-09_16-06-30_rates.png
2022-03-09_16-06-30_rates.png
Attachment 10: 2022-03-09_16-07-43_pn_waves.png
2022-03-09_16-07-43_pn_waves.png
Attachment 11: 2022-03-09_16-08-12_nn_waves.png
2022-03-09_16-08-12_nn_waves.png
Attachment 12: 2022-03-09_16-10-10_pn_widths.png
2022-03-09_16-10-10_pn_widths.png
Attachment 13: 2022-03-09_16-11-14_nn_widths.png
2022-03-09_16-11-14_nn_widths.png
Attachment 14: 2022-03-09_16-40-33_stats.png
2022-03-09_16-40-33_stats.png
Attachment 15: 2022-03-09_16-42-44_rates.png
2022-03-09_16-42-44_rates.png
Attachment 16: 2022-03-09_16-43-46_pn_widths.png
2022-03-09_16-43-46_pn_widths.png
Attachment 17: 2022-03-09_16-44-46_nn_widths.png
2022-03-09_16-44-46_nn_widths.png
Attachment 18: 2022-03-09_16-45-39_pn_waves.png
2022-03-09_16-45-39_pn_waves.png
Attachment 19: 2022-03-09_16-46-08_nn_waves.png
2022-03-09_16-46-08_nn_waves.png
Attachment 20: 2022-03-09_17-03-33_stats.png
2022-03-09_17-03-33_stats.png
Attachment 21: 2022-03-09_17-04-22_rates.png
2022-03-09_17-04-22_rates.png
Attachment 22: 2022-03-09_17-07-44_nn_widths.png
2022-03-09_17-07-44_nn_widths.png
Attachment 23: 2022-03-09_17-06-40_pn_widths.png
2022-03-09_17-06-40_pn_widths.png
Attachment 24: 2022-03-09_17-05-41_nn_waves.png
2022-03-09_17-05-41_nn_waves.png
Attachment 25: 2022-03-09_17-05-11_pn_waves.png
2022-03-09_17-05-11_pn_waves.png
Attachment 26: 2022-03-09_17-26-03_nn_widths.png
2022-03-09_17-26-03_nn_widths.png
Attachment 27: 2022-03-09_17-24-58_pn_widths.png
2022-03-09_17-24-58_pn_widths.png
Attachment 28: 2022-03-09_17-23-57_stats.png
2022-03-09_17-23-57_stats.png
Attachment 29: 2022-03-09_17-43-33nn_widths.png
2022-03-09_17-43-33nn_widths.png
Attachment 30: 2022-03-09_17-42-29_pn_widths.png
2022-03-09_17-42-29_pn_widths.png
Attachment 31: 2022-03-09_17-41-23_stats.png
2022-03-09_17-41-23_stats.png
Attachment 32: 2022-03-09_17-45-55_nn_waves.png
2022-03-09_17-45-55_nn_waves.png
Attachment 33: 2022-03-09_17-45-27_pn_waves.png
2022-03-09_17-45-27_pn_waves.png
Attachment 34: Screenshot_from_2022-03-09_18-12-13.png
Screenshot_from_2022-03-09_18-12-13.png
Attachment 35: Screenshot_from_2022-03-09_18-11-32.png
Screenshot_from_2022-03-09_18-11-32.png
Attachment 36: Screenshot_from_2022-03-09_18-27-23.png
Screenshot_from_2022-03-09_18-27-23.png
Attachment 37: Screenshot_from_2022-03-09_18-32-13.png
Screenshot_from_2022-03-09_18-32-13.png
Attachment 38: Screenshot_from_2022-03-09_18-35-24.png
Screenshot_from_2022-03-09_18-35-24.png
  405   Tue Mar 8 10:26:56 2022 TD NHTuesday 8 March 2022
11.25 System wide checks OK except aida06 clock fail status 6 - attachments 1-13

      ADC data item stats - attachment 14

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage currents - attachment 16
  
      DSSSD#1 OK. DSSSD#2 leakage current has increased from c.5uA to c. 11uA overnight.


11.43 per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - attachments 17-24

      per FEE64 1.8.L spectra - attachments 25-26
          aida12 1.8.L pulser peak width 143 ch FWHM
          aida02 1.8.L pulser peak width 439 ch FWHM

      Per FEE64 rate spectra - attachment 27

12.00 Cycled DSSSD#2 bias from -100V to 0 to -100V leakage current -7.7uA but ramping slowly (c. 5nA/10s ... ?)

13.26 DSSSD#2 bias -100V leakage current -8.54uA

15.46 - Noticed that DSSSD#1 braid no longer has filter, DSSSD#1 core still does. DSSSD#2 is unfiltered on both.

      - Swap FEE64 power connectors #3 with #4, and #6 with #7, this means both n+n FEE64s are on the same paired channel

      No significant change to stats

      - Notice braid was only connected to FEEs 4 and 8, which meant beam-left bias guard ring 0v was floating. Use T to bring bias braid over to FEES 2 and 6 as well.
        Middle bias guard ring 0v is still floating

      No significant change to stats, amplitude of 100 kHz waveform in n+n waves seems attenuated though

     aida12 1.8.L pulser peak width 76 ch FWHM
     aida02 1.8.L pulser peak width 438 ch FWHM
 
     Attachment 28 - HV leakage current
     Attachment 29 - FEE ADC stats
     Attachment 30 - p+n side waveforms
     Attachment 31 - n+n side waveforms
     Attachment 32 - p+n pulser peak widths
     Attachment 33 - n+n pulser peak widths

15.55 - Reattach filter to DSSSD#1 braid

     No major change, possible improvement in waveforms 
     Attachment 34 - Stats
     Attachment 35,36 - waveforms

16.46 - Reattach LK1 jumper to FEES 2, 4, 6, 8

     Waveforms look OK, p+n rate slightly worse?
     Attachment 37 - Stats
     Attachment 38, 39 - waveforms
     Attachment 40 - Per FEE Rates
Attachment 1: Screenshot_from_2022-03-08_11-25-00.png
Screenshot_from_2022-03-08_11-25-00.png
Attachment 2: Screenshot_from_2022-03-08_11-24-55.png
Screenshot_from_2022-03-08_11-24-55.png
Attachment 3: Screenshot_from_2022-03-08_11-24-48.png
Screenshot_from_2022-03-08_11-24-48.png
Attachment 4: Screenshot_from_2022-03-08_11-24-42.png
Screenshot_from_2022-03-08_11-24-42.png
Attachment 5: Screenshot_from_2022-03-08_11-24-35.png
Screenshot_from_2022-03-08_11-24-35.png
Attachment 6: Screenshot_from_2022-03-08_11-24-30.png
Screenshot_from_2022-03-08_11-24-30.png
Attachment 7: Screenshot_from_2022-03-08_11-24-23.png
Screenshot_from_2022-03-08_11-24-23.png
Attachment 8: Screenshot_from_2022-03-08_11-24-15.png
Screenshot_from_2022-03-08_11-24-15.png
Attachment 9: Screenshot_from_2022-03-08_11-23-09.png
Screenshot_from_2022-03-08_11-23-09.png
Attachment 10: Screenshot_from_2022-03-08_11-22-45.png
Screenshot_from_2022-03-08_11-22-45.png
Attachment 11: Screenshot_from_2022-03-08_11-22-36.png
Screenshot_from_2022-03-08_11-22-36.png
Attachment 12: Screenshot_from_2022-03-08_11-21-52.png
Screenshot_from_2022-03-08_11-21-52.png
Attachment 13: Screenshot_from_2022-03-08_11-21-41.png
Screenshot_from_2022-03-08_11-21-41.png
Attachment 14: Screenshot_from_2022-03-08_11-21-30.png
Screenshot_from_2022-03-08_11-21-30.png
Attachment 15: Screenshot_from_2022-03-08_11-21-20.png
Screenshot_from_2022-03-08_11-21-20.png
Attachment 16: Screenshot_from_2022-03-08_11-21-02.png
Screenshot_from_2022-03-08_11-21-02.png
Attachment 17: Screenshot_from_2022-03-08_11-40-39.png
Screenshot_from_2022-03-08_11-40-39.png
Attachment 18: Screenshot_from_2022-03-08_11-40-03.png
Screenshot_from_2022-03-08_11-40-03.png
Attachment 19: Screenshot_from_2022-03-08_11-39-35.png
Screenshot_from_2022-03-08_11-39-35.png
Attachment 20: Screenshot_from_2022-03-08_11-39-14.png
Screenshot_from_2022-03-08_11-39-14.png
Attachment 21: Screenshot_from_2022-03-08_11-38-07.png
Screenshot_from_2022-03-08_11-38-07.png
Attachment 22: Screenshot_from_2022-03-08_11-37-33.png
Screenshot_from_2022-03-08_11-37-33.png
Attachment 23: Screenshot_from_2022-03-08_11-37-14.png
Screenshot_from_2022-03-08_11-37-14.png
Attachment 24: Screenshot_from_2022-03-08_11-35-51.png
Screenshot_from_2022-03-08_11-35-51.png
Attachment 25: Screenshot_from_2022-03-08_11-35-16.png
Screenshot_from_2022-03-08_11-35-16.png
Attachment 26: Screenshot_from_2022-03-08_11-32-45.png
Screenshot_from_2022-03-08_11-32-45.png
Attachment 27: Screenshot_from_2022-03-08_11-30-24.png
Screenshot_from_2022-03-08_11-30-24.png
Attachment 28: 2022-03-08_15-48-40_HV.png
2022-03-08_15-48-40_HV.png
Attachment 29: 2022-03-08_15-49-03_stats.png
2022-03-08_15-49-03_stats.png
Attachment 30: 2022-03-08_15-53-42_pn_waves.png
2022-03-08_15-53-42_pn_waves.png
Attachment 31: 2022-03-08_15-50-12_nn_waves.png
2022-03-08_15-50-12_nn_waves.png
Attachment 32: 2022-03-08_15-54-40_pn_widths.png
2022-03-08_15-54-40_pn_widths.png
Attachment 33: 2022-03-08_15-57-00_nn_widths.png
2022-03-08_15-57-00_nn_widths.png
Attachment 34: 2022-03-08_16-13-21_stats.png
2022-03-08_16-13-21_stats.png
Attachment 35: 2022-03-08_16-15-35_pn_waves.png
2022-03-08_16-15-35_pn_waves.png
Attachment 36: 2022-03-08_16-14-58_nn_waves.png
2022-03-08_16-14-58_nn_waves.png
Attachment 37: Screenshot_from_2022-03-08_16-39-44.png
Screenshot_from_2022-03-08_16-39-44.png
Attachment 38: Screenshot_from_2022-03-08_16-40-15.png
Screenshot_from_2022-03-08_16-40-15.png
Attachment 39: Screenshot_from_2022-03-08_16-41-08.png
Screenshot_from_2022-03-08_16-41-08.png
Attachment 40: Screenshot_from_2022-03-08_16-41-57.png
Screenshot_from_2022-03-08_16-41-57.png
  404   Mon Mar 7 12:20:26 2022 NH, TDAIDA Grounding Checks Mon 7 Mar
Remove all DSSD ribbon cable braid ground from FEE64s to check n+n ground loops

Removing all on n+n side only and test resistance to ground: 4, 6 and 8 show continuity to ground, 2 shows no connection (braid ring terminal to copper bar)

Remove all p+n braids as wll and check:
- See no continuity to ground (good)
- Snout has no continuity to ground (good)
- n+n side fees 4 and 8 to seem to have continuity to snout though (bad)
- p+n checked and don't seem to have continuity to gnd (except when braid was touching a LEMO barrel)
- n+n aida06 maybe has high resistance connection to ground (bad) or (about 0.7 V drop according to RS multimeter)

Bias+Power up and check noise
aida09 Peak width = 86.03+/- 1.721
aida02 Peak width = 509.48+/- 11.1812

Low frequency noise on wavesforms corresponds to ca 100 kHz - switching power supply?

Attachment 6 - ADC data item stats
all p+n/n+n cable braids disconnected
all FEE64 adaptor PCBS grounded to Copper busbar OK

Remove guard ring jumper on upper-middle p+n FEE (rear guard ring of middle DSSSD)
Attachment 7 - ADC data item states
No real change to rates

Remove field plate jumpers on all p+n FEEs
Attachment 8 - ADC data item states
No real change to rates

Remove bias LK1 jumpers on n+n FEES:
Generally worse across the p+n side, no real change to n+n side

Attachment 9 - Stats with LK1 removed:
aida09 Peak Width = 181.77. p+n see Attachment 10
aida02 Peak width = 505.23. n+n see Attachment 11

Attachment 12 - p+n waveforms
Attachment 13 - n+n waveforms

High frequency component reduced but 100 kHz low frequency component stronger

Remove LK3 from aida03,aida07 (no jumpers at all on any FEE) - DSSSDs floating
No significant change, see attachment 14 stats
Attachment 1: 2022-03-07_14-00_Stats.png
2022-03-07_14-00_Stats.png
Attachment 2: 2022-03-07_14-03-03_pn_widths.png
2022-03-07_14-03-03_pn_widths.png
Attachment 3: 2022-03-07_14-04-13_nn_widths.png
2022-03-07_14-04-13_nn_widths.png
Attachment 4: 022-03-07_14-05-12_pn_waves.png
022-03-07_14-05-12_pn_waves.png
Attachment 5: 2022-03-07_14-05-49_nn_waves.png
2022-03-07_14-05-49_nn_waves.png
Attachment 6: Screenshot_from_2022-03-07_15-28-41.png
Screenshot_from_2022-03-07_15-28-41.png
Attachment 7: 2022-03-07_15-48-02_stats_no_middle_rear_GR.png
2022-03-07_15-48-02_stats_no_middle_rear_GR.png
Attachment 8: 2022-03-07_16-08-11_stats_no_fieldplane_ground.png
2022-03-07_16-08-11_stats_no_fieldplane_ground.png
Attachment 9: 2022-03-07_16-22-00_stats_no_LK1.png
2022-03-07_16-22-00_stats_no_LK1.png
Attachment 10: 2022-03-07_16-23-50_pn_widths_no_lk1.png
2022-03-07_16-23-50_pn_widths_no_lk1.png
Attachment 11: 2022-03-07_16-24-55_nn_widths_no_lk1.png
2022-03-07_16-24-55_nn_widths_no_lk1.png
Attachment 12: 2022-03-07_16-25-50_pn_waves_no_lk1.png
2022-03-07_16-25-50_pn_waves_no_lk1.png
Attachment 13: 2022-03-07_16-27-09_nn_waves_no_lk1.png
2022-03-07_16-27-09_nn_waves_no_lk1.png
Attachment 14: 2022-03-07_17-12-34_no_detector_gnd__stats.png
2022-03-07_17-12-34_no_detector_gnd__stats.png
  403   Sat Mar 5 10:43:45 2022 OH, NHGrounding and noise tests 4th-5th March Summary

Tests performed

  • After leaving overnight noticed the pulser was not fully inserted into aida09
    • Little change in rates observed in 9 but otherwise nothing else
  • Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
    • No change observed in rates
  • Also tested covering the snout in light tight material
    • No Changes
  • Unplugged the system monitor from 6 and 8, no change
  • Connected the snout to ground with copper tape
    • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
    • Removed the ground and powercycled and rates restored to where previously
  • Changed the bias supply jumper inside to have the bias line connected to the rack ground
    • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
      • Recrimped and inserted again
    • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
    • Pulser widths however larger than yesterday

p+n width

FEE Width
9 118
1 88
10 75
11 77
3 67
12 66
13 96
5 89
14 107
15 93
7 118
16 87

n+n width

  • Connected up entire pulser circuit to n+n FEEs
FEE Width
2 428
4 N/A
6 428
8 343

Switching bias supply back

  • because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
  • DAQ was then left overnight as had to leave to catch the bus
  • Rates are observed to be back at their higher values - attachment 2
  • n+n 1.8.L spectra - attachment 3
  • p+n 1.8.L spectra - attachment 4

p+n widths

FEE Wdith
9 130
1 128
10 75
11 78
3 71
12 71
13 96
5 85
14 110
15 99
7 123 Dbl
16 92

n+n widths

FEE width
2 56
4 983
6 523
8 358
  • The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

FEE Width
9 106
1 79
10 72
11 78
3 70
12 73
13 102
5 112
14 105
15 110
7 131
16 104

n+n

FEE Width
2 -
4 298
6- -
8 278
  • Don't see a pulser peak in 4 or 2
  • Will replace cable between 4 and 2 and 2 and 6
  • Statistics at 0x64 with pulser on - atachment 5
  • Now see peaks - attachment 6
  • Statistics at 0xa - attachment 7
FEE Width
2 284
4 1089 (Triple)
6 371
8 318

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

  • Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
  • 0x64 stats - attachment 7
  • 0xa statistics - attachment 8
  • p+n 1.8.L - attachment 9
  • p+n waveform - attachment 10
  • n+n waveform - attachment 11
    • Same high frequency component in both
    • 35 channel spacing corresponds to a frequency of ~1.4MHz
FEE p+n Width
9 109
1 118
10 78
11 80
3 70
12 73
13 92
5 85
14 99
15 100
7 123
16 92
FEE n+n Width
2 531
4 1185 (Triple)
6 536
8 376

Next tests

  • Returned braid to FEE 4 and 8
    • Recovered earlier rates and performance
  • Re-added the ground cable to the snout
    • Much worse noise performance across all FEEs
    • p+n rates in 200k
  • Removed the ground from the snout
    • Got really good rates originally - attachment 12
    • Noticed 1 and 5 failed ADC calibration
    • Calibrated them and there rates increased massively - attachment 13
  • Notice that the pulser peak in FEE2 and 6 keeps dropping out
    • As the cables have been changed multiple times thought it could be that an adaptor board was broken
    • 4 seemed the most likely suspect with its high rates and triple peaks
    • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
    • Pulser peaks were still muissing
    • Issue could be then with adaptor board 2 possibly

Conclusions

  • Rates on the FEEs are in a better place now than they were at the start of the week
    • p+n in particular have shown a large amount of improvement
  • There are still however issues with the n+n FEEs
  • I think there is likely an issue with one of the adaptor boards
    • Possible aida02 at the pulser connection point
    • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
    • Would recomend swapping out aida02 as a test next week
  • Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
    • Nic would like to use something other than clingfilm in future
    • One suggestion would be mylar which could be held in place with strips of double sided tape
    • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
  • Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
  • I am at a slight loss of what else to try

 

 

  • I have left the daq running at 0x64 overnight to check for alphas
  • Histograms and statistics zeroed
Attachment 1: 220304_1704_groundedbias_stats.png
220304_1704_groundedbias_stats.png
Attachment 2: 220305_1154_Stats.png
220305_1154_Stats.png
Attachment 3: 220305_1157_nn_18L.png
220305_1157_nn_18L.png
Attachment 4: 220305_1200_pn_18L.png
220305_1200_pn_18L.png
Attachment 5: 220305_1336_0x64Stats.png
220305_1336_0x64Stats.png
Attachment 6: 220305_1341_nn_18L.png
220305_1341_nn_18L.png
Attachment 7: 220305_1342_Stats.png
220305_1342_Stats.png
Attachment 8: 220305_1405_0x64_stats.png
220305_1405_0x64_stats.png
Attachment 9: 220305_1421_pn_18L.png
220305_1421_pn_18L.png
Attachment 10: 220305_1424_pn_waveform.png
220305_1424_pn_waveform.png
Attachment 11: 220305_1429_nn_waveform.png
220305_1429_nn_waveform.png
Attachment 12: 220305_1518_stats.png
220305_1518_stats.png
Attachment 13: 220305_1519_waveform_enabled.png
220305_1519_waveform_enabled.png
  402   Fri Mar 4 08:40:36 2022 OH, NHAIDA Noise and grounding tests 3rd March - Results

Grounding implemented

  • Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
  • Earth bars are then connected to platform ground on AIDA frame
  • Photo of grounding - attachment 1
  • Improvements not as large as expected from previous experience though
    • Particularly in the n+n side where widths still appear awful
  • While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
    • It was noted that it would need changing at some point but continued with it for the time being
  • Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

FEE Width
9 147
1 116
10 92
11 77
3 128
12 72
13 75
5 59
14 75
15 80
7 150
16 75
  • Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
  • 1.8.L spectra - attachment 3

n+n widths

  • Widths still very large
  • Width for 8 is around 500
  • 1.8.L attachment 2

Additional info

  • Statistics - attachment 4

Jumper optimisation

  • Went to our previously found optimal jumper configuration for the trips
  • n+n LK1 only
  • p+n LK 2 and 4 on all FEE
  • p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
  • While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
  • Saw an improvement for most FEEs
FEE Width
9 134
1 83
10 80
11 68
3 60
12 60
13 71
5 60
14 75
15 82
7 97
16 77
  • Unfortunately saw no improvement for the n+n
  • We event tested having the pulser only going into a single FEE (aida08)
  • Still have a width of 622 for that FEE
  • To compare FEEs we have layout 1 - attachment 5
  • Layouot 1 on a common timescale - attachment 6
    • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
    • Many channels are just showing the pulser rate 25Hz
  • p+n  1.8.L attachment 7
  • p+n waveforms attachment 8
  • n+n waveforms attachment 9
  • Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
  • Meanwhile the p+n strips are silent at 0x64 for the most part.
  • DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

  • Test daisy chaining the n+n across to the other n+n strips
    • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
    • All of the jumpers are however grounded
    • Will still test
    • May investigate also trying this on LK3 on adaptors for aida01 and aida03
  • Observe the output of the new sum inverter on the scope to check for variation in amplitude
Attachment 1: PXL_20220303_104253926.jpg
PXL_20220303_104253926.jpg
Attachment 2: 220303_1310_nn_18l.png
220303_1310_nn_18l.png
Attachment 3: 220303_1313_pn_18L.png
220303_1313_pn_18L.png
Attachment 4: 220303_1313_stats.png
220303_1313_stats.png
Attachment 5: 220303_1438_layout1.png
220303_1438_layout1.png
Attachment 6: 220303_1439_layout1common.png
220303_1439_layout1common.png
Attachment 7: 220303_1446_pn_18L.png
220303_1446_pn_18L.png
Attachment 8: 220303_1452_pn_waveform.png
220303_1452_pn_waveform.png
Attachment 9: 220303_1453_nn_waveforms.png
220303_1453_nn_waveforms.png
  401   Thu Mar 3 08:45:03 2022 OH, NHAIDA Noise and grounding tests 3rd March

Pulser spectra

  • Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
  • p+n peak widths averact around 100 channels
  • n+n around 1000 at 0xa threshold
  • n+n 1.8.L spectra - attachment 1
  • p+n 1.8.L spectra attachment 2

p+n peak widths

FEE Width
9 105
1 82
10 71
11 79
3 96
12 72
13 191
5 101
14 129
15 118
7 110
16

100

n+n peak widths

FEE Width
2 1200
4 1179
6 -
8 1813

Daily checks

  • Fee temperatures ok - attachment 3
  • Bias and leakage currents ok - attachment 4
  • Stats spectra with current grounding - attachment 5
    • Current grounding is just a cable going between each adaptor board and its respective FEE
    • Interfee grounding done by power supply and also pulser circuit
  • Rates for each of the asics - attachment 6
  • p+n waveforms - attachment 7
  • n+n waveforms - attachment 8
    • Lots of noise evident on them

Plans for the day

  • Connect FEEs up to the copper earth bars installed
  • Powercycle and check noise levels
  • If n+n remains significantly higher investigate the n+n pulser circuit further
  • Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4
Attachment 1: 220303_0927_nn_18L.png
220303_0927_nn_18L.png
Attachment 2: 220303_0928_pn_18L.png
220303_0928_pn_18L.png
Attachment 3: 220303_0932_TEMP.png
220303_0932_TEMP.png
Attachment 4: 220303_0933_Bias.png
220303_0933_Bias.png
Attachment 5: 220303_0933_Stats.png
220303_0933_Stats.png
Attachment 6: 220303_0936_Layout1.png
220303_0936_Layout1.png
Attachment 7: 220303_0941_pn_waveform.png
220303_0941_pn_waveform.png
Attachment 8: 220303_0942_nn_waveform.png
220303_0942_nn_waveform.png
  400   Wed Mar 2 16:00:01 2022 OH, NHNoise tests - 2nd March
An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain
  399   Wed Mar 2 10:05:13 2022 OHDAQ Status 2nd March 2022
> DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
> Temperature menu is still responsive and all FEEs report a temperature
> Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
> Merger shows no data arriving as expected - Attachment 3
> Merger data buffers for each FEE - attachment 4
> Merger time errors - attachment 5
> 
> DAQ has not yet been restarted in case Patrick wanted to look.

Looking at the first attachment..... aida09 has rebooted. So it is not setup at all.

This has caused the NewMerger to hang .... which surprises me .... as I thought it was designed to spot a bad FEE and continue with the rest.

Yes I will have a look today as I'm in Daresbury.

More later ......
  398   Wed Mar 2 09:08:44 2022 OHDAQ Status 2nd March 2022
DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5

DAQ has not yet been restarted in case Patrick wanted to look.
Attachment 1: 220302_1002_goodevents.png
220302_1002_goodevents.png
Attachment 2: 220302_1002_white_rabbit.png
220302_1002_white_rabbit.png
Attachment 3: 220302_1003_merger.png
220302_1003_merger.png
Attachment 4: 220302_1003_merger_data_buffers.png
220302_1003_merger_data_buffers.png
Attachment 5: 220302_1004_merger_merge_time_errors.png
220302_1004_merger_merge_time_errors.png
  397   Tue Mar 1 08:29:19 2022 OH, NHFindings from the upgrade to centos7 from SL6

Required packages not included as default

  • csh (For running MIDAS scripts)
  • xinetd (Runs the rdate server for the FEEs)
  • Telnet (Communication with the FEEs)
  • dhcp

Configuration changes

  • FEEs uses NFS2 which is no longer enabled by default
    • Must edit config in /etc/nfs.conf to uncomment the following lines:
      • [nfsd]
      • vers2=y
  • /etc/xinetd.d/time-stream must be edited by uncommenting
    • disable        = no

  • Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

    • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

    • Likewise can check status with `systemctl status $SERVICE_NAME`

  • Also need to check the firewall status of the ethernet port connected to the FEEs.

    • Easiest solution is to set the port to be trusted

Optional extras

  • For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
    • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
    • Can also add screenshot to this - Attachment 2
  • Gnome tweaks also allows the number of workspaces to be changed
  • Also recommend setting workspaces to span displays

 

Attachment 1: TapeServer.desktop
[Desktop Entry]
Name=TapeServer
Exec=/MIDAS/Linux/startup/TapeServer
Terminal=true
Icon=/home/npg/Downloads/midas100.gif
Type=Application
Comment=TapeServer for AIDA

Attachment 2: Screenshot_from_2022-03-01_09-40-18.png
Screenshot_from_2022-03-01_09-40-18.png
  396   Wed Feb 16 10:32:52 2022 PJCSAIDA software changes and some suggestions

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

  395   Fri Feb 11 12:09:24 2022 NHFri 11 Feb AIDA Work
Use beam break between FRS experiments to finalise getting AIDA testable for PJCS etc.

USB hubs reconnected - all 16 ttyUSBs seem to be visible now

New inverter installed (Ortec 533 Sum Invert, brand new) - Pulser T connected to input B1 and inverted out B sent to -ve FEEs

VME crate for VETAR2 is powered on so White Rabbit should be available

All grounds reconnected on FEEs

All FEEs connected to pulser chain again

-

Moved interlock to new Al shelf on AIDA rack so it's not kicked
Powered on OK, all 4 lights on in box but AIDA Relay had to water light
Investigation shows the cable between relay and interlock was broken, one wire had come loose from connector
Resoldered and now it works as usual

System powered up OK
All 16 FEEs respond to serial
All 16 FEEs have WR
All 16 FEEs show pulser ADC entries

Seems good!
  393   Tue Feb 1 15:00:28 2022 NHAIDA CentOS 7 Startup
Starting up AIDA on CentOS 7 to confirm things work (based on some CARME experience)

NFS settings:
/etc/exports copied over from old disk
line to old /MIDAS_130718 export removed (commented) to see if necessary to copy over
nfsv2 enabled in nfs.conf
NFS service enabled and started (systemctl enable nfs-server ; systemctl start nfs-server)

DHCP setting:
DHCP was disabled, enabled and started (systemctl enable dhcpd ; systemctl start dhcpd)
Raspberry Pis pick up IP and become responsive again

Firwall setting:
Interface p4p1 (AIDA switch #1) is changed to zone 'trusted' so that FEEs can connect to NFS
(panicked before unable to get RPC port)

See FEEs starting to boot!
Getting both / and /MIDAS successfully (note confirming that old /MIDAS_130718 is not needed)

FEEs 4 and 9 take longer to boot others (observed before, seems to be PSU slots 4/5 in first take a while to provide voltage)
FEE9 seems to panic after mounting root, waiting to see if it reboots itself (no serial)
... Nope but power cycle fixes it

All FEEs online and Reset/Setup completed... Temps OK

Note 1: CentOS 7 uses systemd, monitoring log is best done as
`journalctl -f` as root.
npg not in sudoers (should be done)

Note 2: System used as graphical terminal for Edinburgh double-alpha experiment until Feb. 11
AIDA operated in background and using port forwarding to access MIDAS/Web mostly
UI has weird workspace configuration by default needs changing to better cope with AIDA windows

Note 3:
Only 4 FEEs are producing serial data to the Pi_Monitor
Probably 2 of the USB hubs were knocked lose (from the USB extender cables) when the Fatima/HPGe cables were removed last year
S4 access is limited but will try to reconnect ASAP

Note 4:
AIDA setup unfortunately quite limited in functionality due to other running experiments:
WR is unavailable as VETAR VME crate was powered off
Not all FEEs are connected to pulser or grounded. Some cables were removed during testing last year.
Some ground cables removed and pulser inverter removed as needed at CARME and S4 access is now limited for WASA tests
  392   Fri Oct 29 23:00:04 2021 NHGround cable information
For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 
  391   Fri Oct 29 10:49:01 2021 OHNoise tests - 28th October -29th October
Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24
Attachment 1: 20211028-1300-rates-groundedfees.png
20211028-1300-rates-groundedfees.png
Attachment 2: 20211028-1300-ratespectra.png
20211028-1300-ratespectra.png
Attachment 3: 20211028-1300-ratespectraymax150.png
20211028-1300-ratespectraymax150.png
Attachment 4: 20211028-1300-aida11wave.png
20211028-1300-aida11wave.png
Attachment 5: 20211028-1300-aida02wave.png
20211028-1300-aida02wave.png
Attachment 6: 20211018-1326-rates-psuchange.png
20211018-1326-rates-psuchange.png
Attachment 7: 20211028-1326-ratesspectra-newpsu.png
20211028-1326-ratesspectra-newpsu.png
Attachment 8: 20211028-1326-rates-linymax10-newpsu.png
20211028-1326-rates-linymax10-newpsu.png
Attachment 9: 20211028-1326-wavesaida12-pnpulser-newpsu.png
20211028-1326-wavesaida12-pnpulser-newpsu.png
Attachment 10: 20211028-1326-wavesaida02-newpsupng
Attachment 11: 20211028-1410-ratespulserconnoff.png
20211028-1410-ratespulserconnoff.png
Attachment 12: 20211028-1412-ratespectra-normal.png
20211028-1412-ratespectra-normal.png
Attachment 13: 20211028-1412-ratesspectra-ymax10.png
20211028-1412-ratesspectra-ymax10.png
Attachment 14: 20211028-1413-ratesymax10-pulserONpng
Attachment 15: 20211028-1414-ratesspectra-pulseron.png
20211028-1414-ratesspectra-pulseron.png
Attachment 16: 20211028-1424-pulserpeakspn.png
20211028-1424-pulserpeakspn.png
Attachment 17: 20211028_1449_nn_18L.png
20211028_1449_nn_18L.png
Attachment 18: 20211028_1451_pn_18l.png
20211028_1451_pn_18l.png
Attachment 19: PXL_20211028_121721708.jpg
PXL_20211028_121721708.jpg
Attachment 20: PXL_20211028_121738665.jpg
PXL_20211028_121738665.jpg
Attachment 21: PXL_20211028_121728173.jpg
PXL_20211028_121728173.jpg
Attachment 22: 20211029_1218_stats.png
20211029_1218_stats.png
Attachment 23: 20211029-1400-pnlk1234nnlk124.png
20211029-1400-pnlk1234nnlk124.png
Attachment 24: 20211029-1436-nolk3bar1and3.png
20211029-1436-nolk3bar1and3.png
  390   Wed Oct 27 15:12:50 2021 OH, NHNoise tests - 27th October
 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6
Attachment 1: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
Attachment 2: 271021-dssds-fixed12.png
271021-dssds-fixed12.png
Attachment 3: 271021-waves-dssd-nn.png
271021-waves-dssd-nn.png
Attachment 4: 271021-waveaida10-pndssd.png
271021-waveaida10-pndssd.png
Attachment 5: 271021-waves-aida02-dssdnn-filter.png
271021-waves-aida02-dssdnn-filter.png
Attachment 6: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
  389   Tue Oct 26 16:08:46 2021 NHFEE and Adaptor board groundings
> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.
  388   Thu Oct 21 16:18:51 2021 OHFEE and Adaptor board groundings
During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   
  387   Thu Oct 21 09:33:08 2021 OHNoise tests - 21st Oct 2021
Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.
Attachment 1: 211021_0951_fee8_x8L.png
211021_0951_fee8_x8L.png
Attachment 2: 211021_0953_fee14waveform_zoom.png
211021_0953_fee14waveform_zoom.png
Attachment 3: 211021_0956_fee14_waveform.png
211021_0956_fee14_waveform.png
Attachment 4: 211021_0957_fee8_x8L.png
211021_0957_fee8_x8L.png
Attachment 5: 211021_0959_fee8_waveform_zoom.png
211021_0959_fee8_waveform_zoom.png
  386   Thu Oct 21 09:08:41 2021 OHMIDAS start up messages
Attachment 1: 8015Startup.txt
tidy up
tclsh8.5: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl6376
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "kr.j1xicijsz"
httpd started on port 8015

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl
created namespace ::AIDA
/DataBaseAccessServer
/NetVarService
/SigTaskService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
DataAcquisitionControlClient
RunControlClient loaded
Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl

Attachment 2: 8115startup
tidy up
tclsh8.5_copy2: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl24260
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "qhsbwphtgqqo"
httpd started on port 8115

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
/TapeServer
loading tcl/NewMergerControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl
/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
RunControl loaded
loading Html/NewMerger/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
NewMerge Control loaded
Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl

Attachment 3: tapestart.txt
Tidy up
master(4653): Operation not permitted
master: no process killed
stats: no process killed
driver: no process killed
linkTCP: no process killed
Starting Tape Server

[1] 24297
MIDAS Tape Server: Message logger not contacted.
MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021  12:22:14
MIDAS Tape Server: Unable to change scheduling priority - Permission denied
MIDAS Tape Server: Using default startup
MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205.
MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081936000.
MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration
MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats
MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305                 
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002                
MIDAS Tape Server: Message reporting level = 0x180fff8
MIDAS Tape Server: Message logging level = 0xfff8
MIDAS Tape Server: Tape Server Options = 0x0
MIDAS Tape Server: File device path base = /TapeData
MIDAS Tape Server: Data buffer size = 65536
MIDAS Tape Server: Tape block size = 65536
MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081535000.
MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081a96000.
MIDAS Tape Server: Capabilities restored.
MIDAS Tape Server: Master global area initialised.
MIDAS Tape Server: Stats task has pid 24299
MIDAS Tape Server: Driver process for /dev/file/0 has pid 24300
MIDAS Tape Server: Driver process for /dev/file/1 has pid 24301
MIDAS Tape Server: Driver process for /dev/null/0 has pid 24302
MIDAS Tape Server: Link task 0 has pid 24303
MIDAS Tape Server: Link task 1 has pid 24304
MIDAS Tape Server: Starting the RPC interface
MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205.
MIDAS Tape Server: Entering server loop
MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205.
MIDAS Tape Statistics: Message logger not contacted.
MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021  12:22:20
MIDAS Tape Statistics: Started with args 10205
MIDAS Tape Statistics: Configuration: SHM key=10205
MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Statistics: Shared memory ID is 3
MIDAS Tape Statistics: Shared memory segment located at address 7fb07f300000.
MIDAS Tape Driver (24302): Message logger not contacted.
MIDAS Tape Driver (24302): Message logger not contacted.
MIDAS Tape Driver (24302): Started with args 2 10205
MIDAS Tape Driver (24302): Configuration: driver=2, key=10205.
MIDAS Tape Driver (24302): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24302): Shared memory ID is 3
MIDAS Tape Driver (24302): Shared memory segment located at address 7f183708f000.
MIDAS Tape Driver (24302): Using device /dev/null/0 of type sink.
MIDAS Tape Driver (24300): Message logger not contacted.
MIDAS Tape Driver (24301): Message logger not contacted.
MIDAS Tape Driver (24300): Message logger not contacted.
MIDAS Tape Driver (24301): Message logger not contacted.
MIDAS Tape Driver (24300): Started with args 0 10205
MIDAS Tape Driver (24301): Started with args 1 10205
MIDAS Tape Driver (24301): Configuration: driver=1, key=10205.
MIDAS Tape Driver (24300): Configuration: driver=0, key=10205.
MIDAS Tape Driver (24301): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24300): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24301): Shared memory ID is 3
MIDAS Tape Driver (24300): Shared memory ID is 3
MIDAS Tape Driver (24300): Shared memory segment located at address 7f3b2d8cd000.
MIDAS Tape Driver (24301): Shared memory segment located at address 7ffb5e9ba000.
MIDAS Tape Driver (24300): Using device /dev/file/0 of type file.
MIDAS Tape Driver (24301): Using device /dev/file/1 of type file.
MIDAS Data Link (24304): Message logger not contacted.
MIDAS Data Link (24304): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (24304): Started with args 10205 21001 21002                
MIDAS Data Link (24304): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2
MIDAS Data Link (24304): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (24304): Shared memory ID is 3
MIDAS Data Link (24304): Shared memory segment located at address 7f7452b59000.
MIDAS Data Link (24303): Message logger not contacted.
MIDAS Data Link (24304): Starting the network interface
MIDAS Data Link (24303): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (24303): Started with args 10205 10305                 
MIDAS Data Link (24303): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1
MIDAS Data Link (24303): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (24303): Shared memory ID is 3
MIDAS Data Link (24303): Shared memory segment located at address 7f754106b000.
MIDAS Data Link (24303): Starting the network interface
MIDAS Data Link (24303): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24303): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24303): MIDAS Data Link thread 0 using TCP port 10305.
MIDAS Data Link (24303): Entering server loop
MIDAS Data Link (24303): thread 0 listening on port 10305
MIDAS Data Link (24304): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24304): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24304): MIDAS Data Link thread 0 using TCP port 21001.
MIDAS Data Link (24304): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24304): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24304): MIDAS Data Link thread 1 using TCP port 21002.
MIDAS Data Link (24304): Entering server loop
MIDAS Data Link (24304): thread 0 listening on port 21001
MIDAS Data Link (24304): thread 1 listening on port 21002

Attachment 4: mergerstart.txt
Tidy up
MERGE: no process killed
merge64.AD: no process killed
master64: no process killed
link64: no process killed
merge64: no process killed
statrate64: no process killed
Starting New Merger

[1] 24325
MERGE Master (24325): Message logger not contacted.
MERGE Master (24325): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
Creating NetVars
Creating NetVars complete
MERGE Master (24325): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
MERGE Master (24325): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0
MERGE Master (24325): File mapped object /SHM_11000 of size 536628 created
MERGE Master (24325): Memory mapping 536628 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5efe67a000.
MERGE Master (24325): File mapped object /SHM_11001 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ef8678000.
MERGE Master (24325): File mapped object /SHM_11002 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ef2676000.
MERGE Master (24325): File mapped object /SHM_11003 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eec674000.
MERGE Master (24325): File mapped object /SHM_11004 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ee6672000.
MERGE Master (24325): File mapped object /SHM_11005 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ee0670000.
MERGE Master (24325): File mapped object /SHM_11006 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eda66e000.
MERGE Master (24325): File mapped object /SHM_11007 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ed466c000.
MERGE Master (24325): File mapped object /SHM_11008 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ece66a000.
MERGE Master (24325): File mapped object /SHM_11009 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ec8668000.
MERGE Master (24325): File mapped object /SHM_11010 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ec2666000.
MERGE Master (24325): File mapped object /SHM_11011 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ebc664000.
MERGE Master (24325): File mapped object /SHM_11012 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eb6662000.
MERGE Master (24325): File mapped object /SHM_11013 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eb0660000.
MERGE Master (24325): File mapped object /SHM_11014 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eaa65e000.
MERGE Master (24325): File mapped object /SHM_11015 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ea465c000.
MERGE Master (24325): File mapped object /SHM_11016 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5e9e65a000.
MERGE Master (24325): Shared memory segment located at address 0x7f5e9e65a000.
MERGE Master (24325): Starting stats/rates task
MERGE Master (24325): Stats/Rates task has pid 24327
MERGE Master (24325): Starting 16 link tasks
MERGE Master (24325): Link task has pid 24328
MERGE Master (24325): Link task has pid 24329
MERGE Master (24325): Link task has pid 24330
MERGE Master (24325): Link task has pid 24331
MERGE Master (24325): Link task has pid 24332
MERGE Master (24325): Link task has pid 24333
MERGE Master (24325): Link task has pid 24334
MERGE Master (24325): Link task has pid 24335
MERGE Master (24325): Link task has pid 24336
MERGE Master (24325): Link task has pid 24337
MERGE Statistics/Rates (24327): Message logger not contacted.
MERGE Statistics/Rates (24327): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
Creating NetVars
Creating NetVars complete
MERGE Master (24325): Link task has pid 24338
MERGE Master (24325): Link task has pid 24339
MERGE Master (24325): Link task has pid 24340
MERGE Master (24325): Link task has pid 24341
MERGE Master (24325): Link task has pid 24342
MERGE Master (24325): Link task has pid 24343
MERGE Data Link (24329): Message logger not contacted.
MERGE Data Link (24330): Message logger not contacted.
MERGE Data Link (24330): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24329): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
Creating NetVars complete
MERGE Data Link (24330): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0
MERGE Data Link (24330): File mapped object /SHM_11000 accessed
MERGE Data Link (24330): Memory mapping 536628 bytes
MERGE Data Link (24330): Shared memory segment located at address 0x7f6556fb6000.
MERGE Data Link (24330): Allocated 71720 bytes for local input buffer
MERGE Data Link (24330): File mapped object /SHM_11003 accessed
MERGE Data Link (24330): Memory mapping 100669480 bytes
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24328): Message logger not contacted.
MERGE Data Link (24328): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24328): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0
MERGE Data Link (24328): File mapped object /SHM_11000 accessed
MERGE Data Link (24328): Memory mapping 536628 bytes
MERGE Data Link (24328): Shared memory segment located at address 0x7efe94f4e000.
MERGE Data Link (24328): Allocated 71720 bytes for local input buffer
MERGE Data Link (24328): File mapped object /SHM_11001 accessed
MERGE Data Link (24328): Memory mapping 100669480 bytes
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24336): Message logger not contacted.
MERGE Data Link (24336): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24336): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0
MERGE Data Link (24336): File mapped object /SHM_11000 accessed
MERGE Data Link (24336): Memory mapping 536628 bytes
MERGE Data Link (24336): Shared memory segment located at address 0x7f020eeee000.
MERGE Data Link (24336): Allocated 71720 bytes for local input buffer
MERGE Data Link (24336): File mapped object /SHM_11009 accessed
MERGE Data Link (24336): Memory mapping 100669480 bytes
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24331): Message logger not contacted.
MERGE Data Link (24331): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24331): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0
MERGE Data Link (24331): File mapped object /SHM_11000 accessed
MERGE Data Link (24331): Memory mapping 536628 bytes
MERGE Data Link (24331): Shared memory segment located at address 0x7f44daa69000.
MERGE Data Link (24331): Allocated 71720 bytes for local input buffer
MERGE Data Link (24331): File mapped object /SHM_11004 accessed
MERGE Data Link (24331): Memory mapping 100669480 bytes
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24337): Message logger not contacted.
MERGE Data Link (24337): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24334): Message logger not contacted.
MERGE Data Link (24334): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24343): Message logger not contacted.
Creating NetVars
MERGE Data Link (24343): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24333): Message logger not contacted.
MERGE Data Link (24333): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24335): Message logger not contacted.
MERGE Data Link (24342): Message logger not contacted.
MERGE Data Link (24342): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24335): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
MERGE Data Link (24341): Message logger not contacted.
Creating NetVars complete
MERGE Data Link (24337): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0
MERGE Data Link (24337): File mapped object /SHM_11000 accessed
MERGE Data Link (24341): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24337): Memory mapping 536628 bytes
MERGE Data Link (24337): Shared memory segment located at address 0x7f12856cb000.
Creating NetVars
Creating NetVars complete
MERGE Data Link (24334): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0
MERGE Data Link (24334): File mapped object /SHM_11000 accessed
MERGE Data Link (24334): Memory mapping 536628 bytes
MERGE Data Link (24332): Message logger not contacted.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb1348e4000.
MERGE Data Link (24337): Allocated 71720 bytes for local input buffer
MERGE Data Link (24337): File mapped object /SHM_11010 accessed
MERGE Data Link (24332): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24337): Memory mapping 100669480 bytes
MERGE Data Link (24340): Message logger not contacted.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
Creating NetVars
MERGE Data Link (24340): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24334): Allocated 71720 bytes for local input buffer
Creating NetVars
MERGE Data Link (24334): File mapped object /SHM_11007 accessed
MERGE Data Link (24334): Memory mapping 100669480 bytes
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24339): Message logger not contacted.
MERGE Data Link (24339): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24333): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0
Creating NetVars complete
MERGE Data Link (24333): File mapped object /SHM_11000 accessed
MERGE Data Link (24333): Memory mapping 536628 bytes
MERGE Data Link (24335): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a30878000.
MERGE Data Link (24335): File mapped object /SHM_11000 accessed
MERGE Data Link (24335): Memory mapping 536628 bytes
MERGE Data Link (24335): Shared memory segment located at address 0x7fd567c53000.
MERGE Data Link (24333): Allocated 71720 bytes for local input buffer
MERGE Data Link (24335): Allocated 71720 bytes for local input buffer
MERGE Data Link (24335): File mapped object /SHM_11008 accessed
MERGE Data Link (24335): Memory mapping 100669480 bytes
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24333): File mapped object /SHM_11006 accessed
MERGE Data Link (24333): Memory mapping 100669480 bytes
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.
Creating NetVars complete
MERGE Data Link (24341): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0
MERGE Data Link (24341): File mapped object /SHM_11000 accessed
MERGE Data Link (24341): Memory mapping 536628 bytes
MERGE Data Link (24341): Shared memory segment located at address 0x7f84b1fef000.
MERGE Data Link (24341): Allocated 71720 bytes for local input buffer
MERGE Data Link (24341): File mapped object /SHM_11014 accessed
MERGE Data Link (24341): Memory mapping 100669480 bytes
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
Creating NetVars complete
Creating NetVars complete
MERGE Data Link (24332): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0
MERGE Data Link (24340): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0
MERGE Data Link (24332): File mapped object /SHM_11000 accessed
MERGE Data Link (24332): Memory mapping 536628 bytes
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8787db000.
MERGE Data Link (24340): File mapped object /SHM_11000 accessed
MERGE Data Link (24340): Memory mapping 536628 bytes
MERGE Data Link (24340): Shared memory segment located at address 0x7f9523835000.
Creating NetVars complete
MERGE Data Link (24339): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0
MERGE Data Link (24332): Allocated 71720 bytes for local input buffer
MERGE Data Link (24340): Allocated 71720 bytes for local input buffer
MERGE Data Link (24339): File mapped object /SHM_11000 accessed
MERGE Data Link (24332): File mapped object /SHM_11005 accessed
MERGE Data Link (24339): Memory mapping 536628 bytes
MERGE Data Link (24332): Memory mapping 100669480 bytes
MERGE Data Link (24339): Shared memory segment located at address 0x7ff8a2e8c000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24340): File mapped object /SHM_11013 accessed
MERGE Data Link (24340): Memory mapping 100669480 bytes
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24339): Allocated 71720 bytes for local input buffer
MERGE Data Link (24339): File mapped object /SHM_11012 accessed
MERGE Data Link (24339): Memory mapping 100669480 bytes
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24338): Message logger not contacted.
MERGE Data Link (24338): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24338): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0
MERGE Data Link (24338): File mapped object /SHM_11000 accessed
MERGE Data Link (24338): Memory mapping 536628 bytes
MERGE Data Link (24338): Shared memory segment located at address 0x7f62f5191000.
MERGE Data Link (24338): Allocated 71720 bytes for local input buffer
MERGE Data Link (24338): File mapped object /SHM_11011 accessed
MERGE Data Link (24338): Memory mapping 100669480 bytes
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Statistics/Rates (24327): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
MERGE Statistics thread starting
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.

Attachment 5: datarelay.txt
Tidy up
DataRelayMBS: no process killed
Starting Data Relay to MBS

Setting Overlap Mode 0
Setting Blocking Mode 0
Setting nice 1
Setting Endian 1
Setting Transfer Mode 4
Setting Transfer Block Size 65536
TCP transfer library version 4.0
0: TCP socket send buffer was 16384 - now 249856
0: TCP socket receive buffer was 87380 - now 249856
0: TCP socket created OK - now connecting to x86l-94 port 6500
Transfer Error - : Connection refused
Error 111
0: connect() failed:
Installing signal handlers for MBS connection
Unable to set signal handler for 9
Unable to set signal handler for 19
Unable to set signal handler for 32
Unable to set signal handler for 33
sigtrap 0
sigtrap 0
TCP transfer library version 4.0
0: TCP socket send buffer was 16384 - now 249856
0: TCP socket receive buffer was 87380 - now 249856
0: TCP socket created OK - now connecting to x86l-94 port 6500
Transfer Error - : Connection refused
Error 111
0: connect() failed:
Sent Start to MBS with code=-1
sigtrap 0
dataSpy Shared buffer area 0 (/SHM_110205) located at 0x6a586000
dataSpy Current age 0 index 0
Spy open = 0
  385   Wed Oct 20 13:26:08 2021 NH, OHAIDA Noise + Filtering
Turn on AIDA (SL6) to investigate noise situation and impact of new RC filters on bias input

Current setup:
- No Detectors
- Pulser in fee14, fee8 *only*, no bias connected. FEEs isolated from each other

Pulser widths::
p+n grounds connected via bias link (no connection to HV bias itself)

ASICFEE14 WidthFEE8 Width
1.8.L14.4121.86
2.8.L13.0221.87
3.8.L12.5721.52
4.8.L13.7523.59


Note: FEE8 is connected via SA1 inverter -> cause of worse resolution

Plug bias into outer FEEs (14 + 8 + chaint o 6)

ASICFEE14 WidthFEE8 Width
1.8.L14.5924.43
2.8.L12.7322.25
3.8.L12.7623.06
4.8.L14.3623.31


Long run over lunch:

ASICFEE14 WidthFEE8 Width
1.8.L14.4725.52
2.8.L12.8324.26
3.8.L12.7524.15
4.8.L14.5826.93

Figures 1 & 2

Add HV filters

ASICFEE14 WidthFEE8 Width
1.8.L14.3423.53
2.8.L12.6723.82
3.8.L12.7823.59
4.8.L13.8825.76

Figures 3 & 4

Small improvement maybe?

Figs 5&6 - Oscilloscope before & after HV filter added

--

Plans for tomorrow (21/10/21)

- Reintroduce pulser cables through all FEEs to see impact on noise (maybe improved now)
- Measure pulser noise in battery DSO (+ other sourceS?)
- Add a DSSD and see impact on noise with HV filter
- Resistance from FEE64 adapter to mechanical ground
- Check pulser on the battery powered scope (Is the signal common to the power supply)
Attachment 1: 211020_1316_fee14_bias_con_long_stats.png
211020_1316_fee14_bias_con_long_stats.png
Attachment 2: 211020_1314_fee8_bias_con_long_stat.png
211020_1314_fee8_bias_con_long_stat.png
Attachment 3: 211020_1421_fee14_bias_con_filtered_long.png
211020_1421_fee14_bias_con_filtered_long.png
Attachment 4: 211020_1415_fee8_bias_con_filtered_long.png
211020_1415_fee8_bias_con_filtered_long.png
Attachment 5: scope_before.png
scope_before.png
Attachment 6: scope_after.png
scope_after.png
  384   Tue Oct 19 16:17:49 2021 OHAIDA-gsi server to centos 7
Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.
  383   Mon Oct 18 14:20:32 2021 OHTop bar launchers
launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs
  382   Mon Oct 18 13:03:23 2021 OHDHCPd.conf backup
Attachment 1: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }

   host aida13 {
     hardware ethernet d8:80:39:42:d:15;
     fixed-address 192.168.11.13;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   }

   host aida14 {
     hardware ethernet d8:80:39:42:d:b;
     fixed-address 192.168.11.14;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   }

   host aida15 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.15;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15";
   }

   host aida16 {
     hardware ethernet d8:80:39:41:f6:ed;
     fixed-address 192.168.11.16;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida16";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



  381   Mon Oct 18 13:01:04 2021 OHProxy config
Proxy for firefox and anydesk
Attachment 1: proxy.png
proxy.png
Attachment 2: proxy2.png
proxy2.png
  380   Fri Sep 3 13:39:42 2021 OHAdditional noise tests performed at GSI
Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)
Attachment 1: DSO_Lab_NoP.jpg
DSO_Lab_NoP.jpg
Attachment 2: DSO_Lab_1.jpg
DSO_Lab_1.jpg
Attachment 3: DSO_Lab_50V.jpg
DSO_Lab_50V.jpg
Attachment 4: DSO_Lab_NoCable.jpg
DSO_Lab_NoCable.jpg
  379   Sun Aug 22 18:40:08 2021 TDAIDA Noise
> > 
> > 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
> >       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
> >       Switch Cooknell SA1 Sum Amp output from negative to positive
> > 
> >       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> > 
> >       This probably indicates
> >       - noisy Cooknell SA1
> >       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> > 
> > > Investigate noise situation of AIDA (no DSSSDs connected)
> > > 
> > > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > > - back-side aidas (02,04,06,08) worse resolution than front-side
> > > - high-frequency noise still noticeable in waveforms
> > > 
> > > Figs 7-8: Remove pulser in (from NIM rack) only
> > > - no noticable change
> > > 
> > > Figs 9-10: Remove all pulser cables from adapter boards
> > > - looks improved
> > > 
> > > Figs 11-12: HV Braid->GND Jumpers removed
> > > 
> > > Fig 13-14: All HV cables removed
> > > - aida06, aida14 isn't updating
> > > - very noticable improvement
> > > 
> > > Fig 15-16: Intermediate HV cables re-added
> > > - signal worse in FEE64s with HV cables
> > > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > > 
> > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > > 
> > > Pulser widths now:
> > > - aida14 (-ve amplitude) = 13.5 channels
> > > - aida08 (+ve amplitude) = 40 channels (?)
> 
> Removed the T-piece between pulser and summing amplifier
> pulser FWHM still ~30 channels - not the T-piece, probably summing amp
> 
> Checked the noise of HV and LV on battery DSO
> - LV - no noticeable noise on any channel (< 5 mV rms)
> - HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
>        independent of bias voltage 
>        See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
>                       AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2
> 
> Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
> 100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
> Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 

1nF/3k => 333kHz low pass rolloff

10nF/1M => 100Hz low pass rolloff

>  Check on one DSSSD, if see improvement can migrate to adapter boards in new revision
> 
> Things to do:
> TD:
>  Inline filters
> NH:
>  Check same behaviour other N1419ET HV supplies
>  Check line voltages from CAEN NIM bin
  378   Fri Aug 20 14:33:24 2021 TDAIDA Noise
> 
> 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
>       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
>       Switch Cooknell SA1 Sum Amp output from negative to positive
> 
>       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> 
>       This probably indicates
>       - noisy Cooknell SA1
>       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> 
> > Investigate noise situation of AIDA (no DSSSDs connected)
> > 
> > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > - back-side aidas (02,04,06,08) worse resolution than front-side
> > - high-frequency noise still noticeable in waveforms
> > 
> > Figs 7-8: Remove pulser in (from NIM rack) only
> > - no noticable change
> > 
> > Figs 9-10: Remove all pulser cables from adapter boards
> > - looks improved
> > 
> > Figs 11-12: HV Braid->GND Jumpers removed
> > 
> > Fig 13-14: All HV cables removed
> > - aida06, aida14 isn't updating
> > - very noticable improvement
> > 
> > Fig 15-16: Intermediate HV cables re-added
> > - signal worse in FEE64s with HV cables
> > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > 
> > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > 
> > Pulser widths now:
> > - aida14 (-ve amplitude) = 13.5 channels
> > - aida08 (+ve amplitude) = 40 channels (?)

Removed the T-piece between pulser and summing amplifier
pulser FWHM still ~30 channels - not the T-piece, probably summing amp

Checked the noise of HV and LV on battery DSO
- LV - no noticeable noise on any channel (< 5 mV rms)
- HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
       independent of bias voltage 
       See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
                      AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2

Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 
 Check on one DSSSD, if see improvement can migrate to adapter boards in new revision

Things to do:
TD:
 Inline filters
NH:
 Check same behaviour other N1419ET HV supplies
 Check line voltages from CAEN NIM bin
Attachment 1: aida_0v.jpg
aida_0v.jpg
Attachment 2: aida_10v.jpg
aida_10v.jpg
Attachment 3: aida_20v.jpg
aida_20v.jpg
Attachment 4: aida_50v.jpg
aida_50v.jpg
Attachment 5: aida_100v.jpg
aida_100v.jpg
Attachment 6: aida_0v.jpg
aida_0v.jpg
  377   Thu Aug 19 10:47:13 2021 TDMIDAS spectra
Attachment 1: AIDA_spectra_190821.tar.gz
  376   Thu Aug 19 10:37:50 2021 TDAIDA Noise
11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
      Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
      Switch Cooknell SA1 Sum Amp output from negative to positive

      Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14

      This probably indicates
      - noisy Cooknell SA1
      - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1

> Investigate noise situation of AIDA (no DSSSDs connected)
> 
> Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> - back-side aidas (02,04,06,08) worse resolution than front-side
> - high-frequency noise still noticeable in waveforms
> 
> Figs 7-8: Remove pulser in (from NIM rack) only
> - no noticable change
> 
> Figs 9-10: Remove all pulser cables from adapter boards
> - looks improved
> 
> Figs 11-12: HV Braid->GND Jumpers removed
> 
> Fig 13-14: All HV cables removed
> - aida06, aida14 isn't updating
> - very noticable improvement
> 
> Fig 15-16: Intermediate HV cables re-added
> - signal worse in FEE64s with HV cables
> - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> 
> Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> 
> Pulser widths now:
> - aida14 (-ve amplitude) = 13.5 channels
> - aida08 (+ve amplitude) = 40 channels (?)
  375   Tue Aug 17 12:39:25 2021 NH, TDAIDA Noise
Investigate noise situation of AIDA (no DSSSDs connected)

Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms

Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change

Figs 9-10: Remove all pulser cables from adapter boards
- looks improved

Figs 11-12: HV Braid->GND Jumpers removed

Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement

Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good

Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.

Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?)
Attachment 1: Screenshot-Waves1.png
Screenshot-Waves1.png
Attachment 2: Screenshot-Waves2.png
Screenshot-Waves2.png
Attachment 3: Screenshot-Pulsers.png
Screenshot-Pulsers.png
Attachment 4: Screenshot-Width11.png
Screenshot-Width11.png
Attachment 5: Screenshot-PulserWidthBack.png
Screenshot-PulserWidthBack.png
Attachment 6: Screenshot-Width02.png
Screenshot-Width02.png
Attachment 7: Screenshot-PulserInRemov-Front.png
Screenshot-PulserInRemov-Front.png
Attachment 8: Screenshot-PulserInRemov-Back.png
Screenshot-PulserInRemov-Back.png
Attachment 9: PulserAllOut-Front.png
PulserAllOut-Front.png
Attachment 10: PulserAllremov-Back.png
PulserAllremov-Back.png
Attachment 11: HVJumpers-Front.png
HVJumpers-Front.png
Attachment 12: HVJumpers-Back.png
HVJumpers-Back.png
Attachment 13: AllHVRemoved-Front.png
AllHVRemoved-Front.png
Attachment 14: HVout-Back.png
HVout-Back.png
Attachment 15: HVintermediate-Front.png
HVintermediate-Front.png
Attachment 16: HVintermediate-Back.png
HVintermediate-Back.png
  374   Mon Jul 5 14:36:15 2021 NHMonday 5 July
AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.

Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.

FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now
  373   Tue Jun 22 13:01:19 2021 NHAIDA Data on lustre
The following runs are now copied to lustre
If necessary they can be removed from AIDA HDDs for space.
There are also spare HDDs that can (and should) be swapped for the current ones

/lustre/gamma/DESPEC_S452_FILES/AIDA
S452
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58

/lustre/gamma/DESPEC_S460_FILES/AIDA
S460
R1
R2
R4
R5
R7
R9
R11
R13
R14
R16
R20
R21
R23
R24
R25
R26
R27
R30
R31
R32
R33
R34
R35
R36
R38
R40
R45
R46
R48
R50
R51
R52

/lustre/gamma/DESPEC_S496_FILES/AIDA
S496
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58
  372   Mon Jun 21 10:21:50 2021 NHMonday 21 June
11.21 - DAQ continues OK (no storage mode)

11.21 - System checks ok *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d95 : 	 89  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16


			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x9560 : 	 38240  
aida13 fault 	 0x0 : 	 0x15b9 : 	 5561  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     12     14      2      4      4      2      2      2      4      6   : 37636
aida02 :     40      7     15      1      3      4      3      2      2      4      6   : 37800
aida03 :     39      7      9      6      5      3      1      3      2      4      6   : 37860
aida04 :     27      7      5      3      3      4      2      3      2      4      6   : 37908
aida05 :     33      5      5      3      6      4      3      3      2      4      6   : 38364
aida06 :     45     12      3      4      4      2      2      3      4      3      6   : 37828
aida07 :     31     11      5      4      4      2      3      2      2      4      6   : 37540
aida08 :     31      6      8      5      3      4      3      2      2      4      6   : 37772
aida09 :     45      4      9      3      2      3      3      2      2      4      6   : 37572
aida10 :     37     11     10      2      3      4      3      3      3      3      6   : 37260
aida11 :     40      9     11      3      3      4      3      3      1      4      6   : 37304
aida12 :     44     11      6      5      3      4      3      3      1      4      6   : 37320
aida13 :     41     11      9      3      5      3      3      3      1      4      6   : 37292
aida14 :     31     12      4      6      5      3      3      3      3      3      6   : 37276
aida15 :     17      9      7      1      2      2      3      2      1      4      7   : 40348
aida16 :     41     12      5      4      5      3      3      3      1      4      6   : 37268


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


11.23 - figures
    Grafana temp & leakage - attachment 1
    Weather cooler this week in Germany, expect S4 to cool down eventually
    Lost activity monitor - attachment 2
    1.8.W spectra - 20us FSR - attachments 3 & 4
    aida05 looks very noisy (?) 
    ADC data items - attachment 5
    FEE64 temperatures OK - attachment 6
    DSSSD bias & leakage currents - attachment 7
    Merger/Tape Server/Merger statistics - attachment 8
    Looks some new time seq error since Thursday

Experiment is runnign in S4 so no access possible this week

Data is being archived to lustre (GSI PB file storage) 
Attachment 1: firefox_2021-06-21_11-27-00.png
firefox_2021-06-21_11-27-00.png
Attachment 2: AnyDesk_2021-06-21_11-24-28.png
AnyDesk_2021-06-21_11-24-28.png
Attachment 3: AnyDesk_2021-06-21_11-25-44.png
AnyDesk_2021-06-21_11-25-44.png
Attachment 4: AnyDesk_2021-06-21_11-26-07.png
AnyDesk_2021-06-21_11-26-07.png
Attachment 5: AnyDesk_2021-06-21_11-24-15.png
AnyDesk_2021-06-21_11-24-15.png
Attachment 6: AnyDesk_2021-06-21_11-24-43.png
AnyDesk_2021-06-21_11-24-43.png
Attachment 7: AnyDesk_2021-06-21_11-25-04.png
AnyDesk_2021-06-21_11-25-04.png
Attachment 8: AnyDesk_2021-06-21_11-25-15.png
AnyDesk_2021-06-21_11-25-15.png
  371   Thu Jun 17 11:14:04 2021 TDThursday 17 June
12.15 DAQ continues OK - no storage mode

12.15 System checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d6 : 	 13014  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     31     11      9      4      4      4      2      2      2      4      6   : 37604
aida02 :     30     10     10      3      3      4      3      2      2      4      6   : 37768
aida03 :     35     11      8      6      5      3      2      2      3      3      6   : 36580
aida04 :     33      9      4      4      3      4      2      3      3      3      6   : 36940
aida05 :     41      9      7      3      4      4      2      2      2      4      6   : 37564
aida06 :     38      6      5      3      2      3      2      3      4      3      6   : 37752
aida07 :     25     12      6      5      4      2      3      2      2      4      6   : 37572
aida08 :     32      7     10      5      4      3      3      2      2      4      6   : 37752
aida09 :     41      8      6      4      2      3      3      2      2      4      6   : 37572
aida10 :     31     10     11      5      5      3      3      3      3      3      6   : 37340
aida11 :     29      7     12      3      4      4      3      3      1      4      6   : 37324
aida12 :     42      8      7      5      3      4      3      3      1      4      6   : 37304
aida13 :     43     10      4      5      5      3      3      3      1      4      6   : 37276
aida14 :     40     15      4      5      4      4      3      3      3      3      6   : 37368
aida15 :     13      5      8      2      1      2      3      2      1      4      7   : 40284
aida16 :     39      9      6      6      5      3      3      3      1      4      6   : 37316

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

12.15 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png
Screenshot_2021-06-17_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1008.png
1008.png
Attachment 8: 1009.png
1009.png
  370   Wed Jun 16 18:05:03 2021 TDWednesday 16 June
19.04 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8960 : 	 28  
aida02 fault 	 0x9b36 : 	 0x9b52 : 	 28  
aida03 fault 	 0xf26b : 	 0xf287 : 	 28  
aida04 fault 	 0x57a0 : 	 0x57bc : 	 28  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d4 : 	 13012  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     53     10      4      4      4      4      2      2      3      3      6   : 36580
aida02 :     28      7     10      4      3      4      3      2      2      4      6   : 37768
aida03 :     41     10      7      6      5      3      1      3      3      3      6   : 36836
aida04 :     45      7      2      4      3      4      2      3      3      3      6   : 36940
aida05 :     35      8      8      4      4      4      2      2      2      4      6   : 37580
aida06 :     45      6      6      3      3      3      2      3      4      3      6   : 37860
aida07 :     37     10      6      4      5      2      3      2      2      4      6   : 37636
aida08 :     32      5     10      4      4      4      3      2      2      4      6   : 37832
aida09 :     24      5      8      7      1      3      3      2      2      4      6   : 37544
aida10 :     29      9     12      5      4      3      3      3      3      3      6   : 37276
aida11 :     33      7      7      6      3      4      3      3      1      4      6   : 37292
aida12 :     36     11      6      6      3      4      3      3      1      4      6   : 37320
aida13 :     37      7      6      6      5      3      3      3      1      4      6   : 37292
aida14 :     25      9      7      6      5      3      3      3      3      3      6   : 37276
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     35      7      6      6      4      4      3      3      1      4      6   : 37348

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.07 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-16_Statistics_aidas-gsi(1).png
Screenshot_2021-06-16_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 1005.png
1005.png
Attachment 7: 1006.png
1006.png
  369   Tue Jun 15 12:35:48 2021 TDTuesday 15 June
13.35 DAQ continues OK - no storage mode

      All system wide checks OK *except*
	
		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     37     10      8      5      4      4      2      2      2      4      6   : 37636
aida02 :     37     10     10      3      4      4      3      2      2      4      6   : 37860
aida03 :     41      8      7      7      5      3      1      3      2      4      6   : 37876
aida04 :     33      9      4      4      2      5      2      3      3      3      6   : 37004
aida05 :     37      9      9      3      5      4      2      2      2      4      6   : 37644
aida06 :     39      9      6      5      4      2      2      3      4      3      6   : 37860
aida07 :     29     12      6      5      4      2      3      2      2      4      6   : 37588
aida08 :     25      7      9      5      4      3      3      2      2      4      6   : 37708
aida09 :     34      5      9      6      2      3      3      2      2      4      6   : 37632
aida10 :     34     12     11      5      4      3      3      3      3      3      6   : 37304
aida11 :     38      8      7      6      3      4      3      3      1      4      6   : 37320
aida12 :     34     10      8      6      3      4      3      3      1      4      6   : 37336
aida13 :     33     11      5      6      4      4      3      3      1      4      6   : 37356
aida14 :     31     10      5      6      5      3      3      3      3      3      6   : 37276
aida15 :     14      6      7      4      1      2      3      2      1      4      7   : 40344
aida16 :     33      8      6      7      5      3      3      3      1      4      6   : 37316

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png
Screenshot_2021-06-15_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1002.png
1002.png
Attachment 8: 1003.png
1003.png
  368   Tue Jun 15 12:33:46 2021 TDMonday 14 June
  367   Sun Jun 13 17:29:04 2021 TDSunday 13 June
18.15 DAQ continues file S496/R32_946
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      Out of disk space on /TapeData filesystem - switch to no storage mode

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     16      4      6      4      4      2      2      2      4      6   : 37608
aida02 :     29     11      9      3      3      4      3      2      2      4      6   : 37756
aida03 :     32      9      6      8      5      3      1      3      3      3      6   : 36840
aida04 :     32      8      3      4      3      4      2      3      3      3      6   : 36912
aida05 :     36      8      6      4      5      4      2      2      2      4      6   : 37616
aida06 :     25     13      4      3      2      3      2      3      4      3      6   : 37740
aida07 :     28     13      4      5      5      2      3      2      2      4      6   : 37624
aida08 :     29      5      9      4      3      4      3      2      2      4      6   : 37740
aida09 :     41      8      4      7      1      3      3      2      2      4      6   : 37572
aida10 :     45     11      7      5      5      3      3      3      3      3      6   : 37340
aida11 :     33     13      5      6      3      4      3      3      1      4      6   : 37308
aida12 :     39      8      6      6      4      4      3      3      1      4      6   : 37372
aida13 :     28     14      3      6      4      4      3      3      1      4      6   : 37328
aida14 :     30     11      3      6      4      4      3      3      3      3      6   : 37312
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     28      9      5      6      5      3      3      3      1      4      6   : 37256

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 2: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png
Attachment 5: Screenshot_2021-06-13_Statistics_aidas-gsi.png
Screenshot_2021-06-13_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1000.png
1000.png
Attachment 8: 1001.png
1001.png
  366   Sat Jun 12 14:53:56 2021 TDaida15 - system console output - 15.33 Sat 12 June reboot
04/11:03:40|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 
04/11:03:41|Last Data : Databuffer  273286648 : 0 => 80500240 : 1 => 0AFF8000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:42|Machine check in kernel mode.
12/15:30:08|I-Cache Parity Error
12/15:30:08|Oops: Machine check, sig: 7 [#1]
12/15:30:08|PREEMPT Xilinx Virtex440
12/15:30:08|Modules linked in: aidamem xdriver xh_spidev_register
12/15:30:08|NIP: c0055c54 LR: c00558fc CTR: c000ae5c
12/15:30:08|REGS: c6ff9f10 TRAP: 0214   Not tainted  (2.6.31)
12/15:30:08|MSR: 00021000 <ME,CE>  CR: 82000028  XER: 20000000
12/15:30:08|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000
12/15:30:08|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 
12/15:30:08|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 
12/15:30:08|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 
12/15:30:08|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 
12/15:30:08|NIP [c0055c54] update_wall_time+0x40c/0xd3c
12/15:30:09|LR [c00558fc] update_wall_time+0xb4/0xd3c
12/15:30:09|Call Trace:
12/15:30:09|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable)
12/15:30:09|[c6339d90] [c0040b84] do_timer+0x38/0x4c
12/15:30:09|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8
12/15:30:09|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120
12/15:30:09|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c
12/15:30:09|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18
12/15:30:09|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem]
12/15:30:09|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188
12/15:30:09|[c6339f10] [c00982f8] sys_read+0x4c/0x90
12/15:30:09|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c
12/15:30:09|Instruction dump:
12/15:30:09|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 
12/15:30:09|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 
12/15:30:09|Kernel panic - not syncing: Fatal exception in interrupt
12/15:30:09|Call Trace:
12/15:30:09|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
12/15:33:10|
Found 0 errors checking kernel image
12/15:33:11|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
12/15:33:11|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
12/15:33:11|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
12/15:33:11|SPIconfig done now to set up the LMK3200 registers
12/15:33:12|LMK #0 : regInit[0]=0x80000000
12/15:33:12|LMK #0 : regInit[1]=0x10070600
12/15:33:12|LMK #0 : regInit[2]=0x60601
12/15:33:12|LMK #0 : regInit[3]=0x60602
12/15:33:12|LMK #0 : regInit[4]=0x60603
12/15:33:12|LMK #0 : regInit[5]=0x70624
12/15:33:12|LMK #0 : regInit[6]=0x70605
12/15:33:12|LMK #0 : regInit[7]=0x70606
12/15:33:12|LMK #0 : regInit[8]=0x70627
12/15:33:12|LMK #0 : regInit[9]=0x10000908
12/15:33:12|LMK #0 : regInit[10]=0xA0022A09
12/15:33:12|LMK #0 : regInit[11]=0x82800B
12/15:33:12|LMK #0 : regInit[12]=0x28C800D
12/15:33:12|LMK #0 : regInit[13]=0x830020E
12/15:33:12|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
12/15:33:12|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
12/15:33:12|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
ADCs calibrated
12/15:33:13|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
12/15:33:13|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
12/15:33:16|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
12/15:33:23|Using Xilinx Virtex440 machine description
12/15:33:23|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
12/15:33:23|Zone PFN ranges:
12/15:33:23|  DMA      0x00000000 -> 0x00007000
12/15:33:23|  Normal   0x00007000 -> 0x00007000
12/15:33:23|Movable zone start PFN for each node
12/15:33:23|early_node_map[1] active PFN ranges
12/15:33:23|    0: 0x00000000 -> 0x00007000
12/15:33:23|MMU: Allocated 1088 bytes of context maps for 255 contexts
12/15:33:23|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
12/15:33:24|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
12/15:33:24|PID hash table entries: 512 (order: 9, 2048 bytes)
12/15:33:24|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
12/15:33:24|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
12/15:33:24|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
12/15:33:24|Kernel virtual memory layout:
12/15:33:24|  * 0xffffe000..0xfffff000  : fixmap
12/15:33:24|  * 0xfde00000..0xfe000000  : consistent mem
12/15:33:24|  * 0xfde00000..0xfde00000  : early ioremap
12/15:33:24|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
12/15:33:24|NR_IRQS:512
12/15:33:24|clocksource: timebase mult[a00000] shift[22] registered
12/15:33:24|Console: colour dummy device 80x25
12/15:33:24|Mount-cache hash table entries: 512
12/15:33:24|NET: Registered protocol family 16
12/15:33:24|PCI: Probing PCI hardware
12/15:33:24|bio: create slab <bio-0> at 0
12/15:33:24|NET: Registered protocol family 2
12/15:33:24|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
12/15:33:24|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
12/15:33:25|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
12/15:33:25|TCP: Hash tables configured (established 4096 bind 4096)
12/15:33:25|TCP reno registered
12/15:33:25|NET: Registered protocol family 1
12/15:33:25|ROMFS MTD (C) 2007 Red Hat, Inc.
12/15:33:25|msgmni has been set to 214
12/15:33:25|io scheduler noop registered
12/15:33:25|io scheduler anticipatory registered
12/15:33:25|io scheduler deadline registered
12/15:33:25|io scheduler cfq registered (default)
12/15:33:25|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
12/15:33:25|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
12/15:33:25|console [ttyS0] enabled
12/15:33:25|brd: module loaded
12/15:33:25|loop: module loaded
12/15:33:25|Device Tree Probing 'ethernet'
12/15:33:25|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
12/15:33:25|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
12/15:33:25|XLlTemac: DCR address: 0x80
12/15:33:25|XLlTemac: buffer descriptor size: 32768 (0x8000)
12/15:33:25|XLlTemac: Allocating DMA descriptors with kmalloc
12/15:33:25|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
12/15:33:26|XTemac: PHY detected at address 7.
12/15:33:26|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
12/15:33:26|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26|Using buffer write method
12/15:33:26|cfi_cmdset_0001: Erase suspend on write enabled
12/15:33:26|cmdlinepart partition parsing not available
12/15:33:26|RedBoot partition parsing not available
  365   Sat Jun 12 13:56:00 2021 TDSaturday 12 June
14.11 DAQ continues file S496/R32_697
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8948 : 	 4  
aida02 fault 	 0x9b36 : 	 0x9b3a : 	 4  
aida03 fault 	 0xf26b : 	 0xf26f : 	 4  
aida04 fault 	 0x57a0 : 	 0x57a4 : 	 4  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x21be : 	 8638  
aida13 fault 	 0x0 : 	 0x2294 : 	 8852  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     10      9      2      5      4      2      2      2      4      6   : 37604
aida02 :     46     12      7      2      4      4      3      2      2      4      6   : 37832
aida03 :     34     10      9      5      4      3      1      3      3      3      6   : 36744
aida04 :     35     12     10      1      3      4      2      3      3      3      6   : 36972
aida05 :     37      9      8      3      4      4      2      2      2      4      6   : 37564
aida06 :     34     10      5      5      4      2      2      3      2      4      6   : 37832
aida07 :     31     11      1      7      4      2      3      2      2      4      6   : 37572
aida08 :     40      7      4      5      3      4      3      2      2      4      6   : 37752
aida09 :     39      9      8      4      1      3      3      2      2      4      6   : 37540
aida10 :     43     12      8      4      4      3      3      3      1      4      6   : 37260
aida11 :     34     14     10      5      3      4      3      3      1      4      6   : 37368
aida12 :     46     12      3      4      4      4      3      3      1      4      6   : 37320
aida13 :     41     11      5      4      5      3      3      3      1      4      6   : 37260
aida14 :     39     10      2      6      4      4      3      3      1      4      6   : 37324
aida15 :     19     12      4      3      2      2      3      2      2      4      6   : 37324
aida16 :     45     10      6      4      5      3      3      3      1      4      6   : 37284

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.05 analysis of file S496/R32_698 - attachment 9
      max dead time 0.05% aida04

15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365

      successful DAQ stop
      aida15 DAQ reset, setup, go 
      successful DAQ go 
Attachment 1: Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-12_Statistics_aidas-gsi.png
Screenshot_2021-06-12_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 998.png
998.png
Attachment 8: 999.png
999.png
Attachment 9: R32_698
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   44970784 (   15685.0 Hz)
        Other data format:  216949222 (   75668.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         62 (       0.0 Hz)
                               RESUME:         62 (       0.0 Hz)
                              SYNC100:      42680 (      14.9 Hz)
                              WR48-63:      42680 (      14.9 Hz)
                           FEE64 disc:     985985 (     343.9 Hz)
                             MBS info:  215877753 (   75294.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2867.115 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2299.418
  2                0.673                0.000
  3                1.368                0.000
  4                0.000                0.000
  5                0.000             2495.101
  6                0.000                0.000
  7                0.000             2566.695
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        467          2          0          0          0          0          1          1          0          0          0          0
  1      72420         30          0          0          0          0         15         15          0          0          0          0
  2        935  107990737          0          0         33         33      19385      19385          0  107951901          0          0
  3   38882631  108955999          0          0         29         29      22052      22052     985985  107925852          0          0
  4        496          0          0          0          0          0          0          0          0          0          0          0
  5    3224207       1336          0          0          0          0        668        668          0          0          0          0
  6       1520          0          0          0          0          0          0          0          0          0          0          0
  7    2784837       1118          0          0          0          0        559        559          0          0          0          0
  8        473          0          0          0          0          0          0          0          0          0          0          0
  9        173          0          0          0          0          0          0          0          0          0          0          0
 10        554          0          0          0          0          0          0          0          0          0          0          0
 11        317          0          0          0          0          0          0          0          0          0          0          0
 12        752          0          0          0          0          0          0          0          0          0          0          0
 13        204          0          0          0          0          0          0          0          0          0          0          0
 14        302          0          0          0          0          0          0          0          0          0          0          0
 15        496          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.945s ( 1068.615 blocks/s,  66.788 Mb/s)
  363   Thu Jun 10 11:32:27 2021 TDThursday 10 June
12.26 DAQ continues file S496/R32_428
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     11      6      3      4      4      2      2      2      4      6   : 37540
aida02 :     30      8      7      3      4      4      3      2      2      4      6   : 37768
aida03 :     38      8     13      3      4      3      1      3      3      3      6   : 36744
aida04 :     37      9      7      2      3      4      2      3      3      3      6   : 36940
aida05 :     29      5      9      3      3      3      1      3      2      4      6   : 37580
aida06 :     38      6      2      3      3      3      2      3      2      4      6   : 37768
aida07 :     31      9      2      7      4      2      3      2      2      4      6   : 37572
aida08 :     34      6      3      5      4      4      3      2      2      4      6   : 37768
aida09 :     29      2     12      5      1      3      3      2      2      4      6   : 37540
aida10 :     24      8     15      3      5      3      3      3      1      4      6   : 37296
aida11 :     35      4     15      3      3      4      3      3      1      4      6   : 37308
aida12 :     36      5      3      5      6      4      3      3      1      4      6   : 37384
aida13 :     31     10      4      3      6      4      3      3      1      4      6   : 37356
aida14 :     37      7      1      4      5      4      3      3      1      4      6   : 37276
aida15 :     34     10      6      3      4      1      2      4      2      4      6   : 38168
aida16 :     31      9      3      3      5      4      3      3      1      4      6   : 37268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 202

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.07 analysis of file S496/R32_431 - attachment 9
      max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png
Screenshot_2021-06-10_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 996.png
996.png
Attachment 8: 997.png
997.png
Attachment 9: R32_431
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   54180182 (   19758.0 Hz)
        Other data format:  207739819 (   75756.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        105 (       0.0 Hz)
                               RESUME:        105 (       0.0 Hz)
                              SYNC100:      42213 (      15.4 Hz)
                              WR48-63:      42213 (      15.4 Hz)
                           FEE64 disc:    1515114 (     552.5 Hz)
                             MBS info:  206140069 (   75173.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          6 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2742.196 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2662.854
  2                0.384                0.000
  3                9.009                0.000
  4                0.000                0.000
  5                0.000             1805.119
  6                0.000                0.000
  7                0.000             1797.783
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        559          0          0          0          0          0          0          0          0          0          0          0
  1     413404        177          0          0          0          0         84         84          9          0          0          0
  2       1153  103268358          0          0         19         19      17963      17963          0  103232394          0          0
  3   42054837  104466440          0          0         86         86      21757      21757    1515079  102907675          0          0
  4        515          0          0          0          0          0          0          0          0          0          0          0
  5    5561752       2359          0          0          0          0       1175       1175          9          0          0          0
  6       1738          2          0          0          0          0          1          1          0          0          0          0
  7    6142236       2472          0          0          0          0       1232       1232          8          0          0          0
  8        565          1          0          0          0          0          0          0          1          0          0          2
  9        190          0          0          0          0          0          0          0          0          0          0          0
 10        694          0          0          0          0          0          0          0          0          0          0          0
 11        417          6          0          0          0          0          0          0          6          0          0          4
 12        967          4          0          0          0          0          1          1          2          0          0          0
 13        244          0          0          0          0          0          0          0          0          0          0          0
 14        285          0          0          0          0          0          0          0          0          0          0          0
 15        626          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.160s ( 1097.388 blocks/s,  68.587 Mb/s)
  362   Wed Jun 9 19:17:21 2021 TDFEE64 system console logs
Attachment 1: logs-090621.tar.gz
  361   Wed Jun 9 11:27:47 2021 TDWednesday 9 June
12.30 DAQ continues OK file S496/R32_373
 
      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5     12      7      2      4      2      2      1      4      7   : 40572
aida02 :     11      6      8      6      1      3      3      2      1      4      7   : 40540
aida03 :     26      5     11      5      1      2      2      3      2      3      7   : 39712
aida04 :     18      7      8      3      2      3      2      3      2      3      7   : 39776
aida05 :     11      5      9      4      1      2      2      3      1      4      7   : 40612
aida06 :     17      3      6      5      5      2      1      3      1      4      7   : 40604
aida07 :      9      3      4      9      1      3      3      2      1      4      7   : 40540
aida08 :     11      8      5      5      3      3      3      2      1      4      7   : 40604
aida09 :     13      5      8      5      2      3      3      2      1      4      7   : 40572
aida10 :     13      8     13      5      1      2      2      2      1      4      7   : 40228
aida11 :     11      5     14      6      3      3      3      3      2      3      7   : 40244
aida12 :     10      9      5      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      7      7      4      3      4      3      3      2      3      7   : 40228
aida14 :      9      6      7      6      3      4      3      3      2      3      7   : 40260
aida15 :     14      3      8      5      1      2      2      4      1      4      7   : 41136
aida16 :     15      4      9      4      4      4      3      3      2      3      7   : 40300

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      1.8.L specttra - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      Lost activity monitor - attachment 4
      1.8.W spectra - 20us FSR - attachments 5 & 6
      ADC data items - attachment 7
      FEE64 temperatures OK - attachment 8
      DSSSD bias & leakage currents OK - attachment 9
      Merger/Tape Server/Merger statistics - attachment 10
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_372 - attachment 11
       max deadtime 0.07% aida04
Attachment 1: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png
Attachment 2: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 3: Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png
Attachment 5: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-06-09_Statistics_aidas-gsi.png
Screenshot_2021-06-09_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: 991.png
991.png
Attachment 10: 992.png
992.png
Attachment 11: R32_372
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   65571173 (   25420.1 Hz)
        Other data format:  196348830 (   76118.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         52 (       0.0 Hz)
                               RESUME:         52 (       0.0 Hz)
                              SYNC100:      41608 (      16.1 Hz)
                              WR48-63:      41608 (      16.1 Hz)
                           FEE64 disc:    2050316 (     794.8 Hz)
                             MBS info:  194215194 (   75291.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        284 (       0.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2579.504 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.015             1853.986
  2                0.548                0.000
  3                1.886                0.000
  4                0.000                0.000
  5                0.000             1457.725
  6                0.000                0.000
  7                0.000             1391.289
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000             1323.904
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000              836.274
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0       6664        532          0          0          0          0          1          1        530          0          0          3
  1    4648488      85466          0          0          1          1        947        947      83570          0          0         92
  2       5675   97165745          0          0         27         27      16487      16487          7   97132710          0          2
  3   47009038   98898439          0          0         24         24      21346      21346    1773239   97082460          0         42
  4       3608          4          0          0          0          0          0          0          4          0          0          1
  5    6949043      95991          0          0          0          0       1383       1383      93225          0          0         60
  6       5001         30          0          0          0          0          1          1          4         24          0          0
  7    6897601     102341          0          0          0          0       1436       1436      99469          0          0         74
  8       1797          1          0          0          0          0          0          0          1          0          0          1
  9       2911          4          0          0          0          0          1          1          2          0          0          3
 10       8722        137          0          0          0          0          1          1        135          0          0          0
 11       5985         36          0          0          0          0          2          2         32          0          0          4
 12       7238         34          0          0          0          0          1          1         32          0          0          0
 13       7345         52          0          0          0          0          0          0         52          0          0          0
 14       4898          4          0          0          0          0          0          0          4          0          0          1
 15       7159         14          0          0          0          0          2          2         10          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.055s ( 1064.726 blocks/s,  66.545 Mb/s)
  360   Tue Jun 8 10:51:55 2021 TDTuesday 8 June
11.52 DAQ continues file S496/R32_340
      High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning

11.53 ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      3     12      8      3      3      2      2      1      4      7   : 40508
aida02 :      5      5     10      6      1      3      3      2      1      4      7   : 40540
aida03 :     25      7     12      5      1      2      2      3      2      3      7   : 39740
aida04 :     19      6     11      5      1      3      2      3      2      3      7   : 39820
aida05 :      9      6      7      5      0      2      2      3      1      4      7   : 40548
aida06 :     10      8      7      4      3      3      1      3      1      4      7   : 40600
aida07 :      9      3      4      8      1      3      3      2      1      4      7   : 40508
aida08 :     12      5      8      5      3      3      3      2      1      4      7   : 40632
aida09 :     15      4      8      5      1      3      3      2      1      4      7   : 40508
aida10 :     23      5     12      5      1      2      2      2      1      4      7   : 40228
aida11 :     15      9     11      6      2      4      3      3      2      3      7   : 40308
aida12 :     14      9      4      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      9      6      3      4      4      3      3      2      3      7   : 40260
aida14 :     11      5      7      6      4      4      3      3      2      3      7   : 40324
aida15 :     16      8      7      4      1      2      2      4      1      4      7   : 41136
aida16 :     15      8      7      4      3      4      3      3      2      3      7   : 40236

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_339 - attachment 9
       max deadtime 0.24% aida04
Attachment 1: Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-08_Statistics_aidas-gsi.png
Screenshot_2021-06-08_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 990.png
990.png
Attachment 8: 991.png
991.png
Attachment 9: R32_339
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   52833088 (   19139.6 Hz)
        Other data format:  209086916 (   75745.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        101 (       0.0 Hz)
                               RESUME:        101 (       0.0 Hz)
                              SYNC100:      42281 (      15.3 Hz)
                              WR48-63:      42281 (      15.3 Hz)
                           FEE64 disc:    1254898 (     454.6 Hz)
                             MBS info:  207747254 (   75259.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2760.405 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2672.516
  2                0.610                0.000
  3                6.711                0.000
  4                0.000                0.000
  5                0.000             2111.274
  6                0.000                0.000
  7                0.000             1910.870
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        589          0          0          0          0          0          0          0          0          0          0          0
  1     221336         98          0          0          0          0         49         49          0          0          0          0
  2       1118  104025022          0          0         30         30      18193      18193          0  103988576          0          0
  3   42352603  105057740          0          0         71         71      22011      22011    1254898  103758678          0          0
  4        548          0          0          0          0          0          0          0          0          0          0          0
  5    4562177       1856          0          0          0          0        928        928          0          0          0          0
  6       1810          0          0          0          0          0          0          0          0          0          0          0
  7    5688898       2196          0          0          0          0       1098       1098          0          0          0          0
  8        593          0          0          0          0          0          0          0          0          0          0          0
  9        215          2          0          0          0          0          1          1          0          0          0          0
 10        675          2          0          0          0          0          1          1          0          0          0          0
 11        387          0          0          0          0          0          0          0          0          0          0          0
 12        964          0          0          0          0          0          0          0          0          0          0          0
 13        267          0          0          0          0          0          0          0          0          0          0          0
 14        369          0          0          0          0          0          0          0          0          0          0          0
 15        539          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.656s ( 1116.685 blocks/s,  69.793 Mb/s)
  359   Mon Jun 7 10:33:48 2021 TDMonday 7 June
11.30 DAQ continues OK - file S496/R32_144
      alpha background

      All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      5     11      6      3      4      2      2      1      4      7   : 40572
aida02 :      1      3      6      7      2      3      3      2      1      4      7   : 40540
aida03 :     21      7      7      8      2      4      1      3      2      3      7   : 39804
aida04 :     13      7      5      6      1      3      2      3      2      3      7   : 39740
aida05 :      2      8      6      5      1      2      2      3      1      4      7   : 40584
aida06 :     10      4      5      5      4      3      1      3      1      4      7   : 40632
aida07 :      9      3      4      7      3      3      3      2      1      4      7   : 40604
aida08 :     11      2      6      4      4      3      3      2      1      4      7   : 40604
aida09 :     10      2      5      4      2      2      2      3      1      4      7   : 40584
aida10 :     14      5      8      6      2      1      1      3      1      4      7   : 40352
aida11 :      8      6      9      5      4      4      3      3      2      3      7   : 40320
aida12 :     10      5      3      7      3      3      2      2      1      4      7   : 40352
aida13 :      7      5      4      6      5      4      3      3      2      3      7   : 40324
aida14 :      7      5      4      8      4      4      3      3      2      3      7   : 40324
aida15 :      8      4      7      4      2      2      2      4      1      4      7   : 41136
aida16 :     13      5      7      6      3      4      3      3      2      3      7   : 40268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

11.35 analysis of file S496/R32_143 - attachment 9
       max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png
Screenshot_2021-06-07_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 986.png
986.png
Attachment 8: 987.png
987.png
Attachment 9: R32_143
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57707988 (   21988.9 Hz)
        Other data format:  204212013 (   77812.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.0 Hz)
                               RESUME:         85 (       0.0 Hz)
                              SYNC100:      41772 (      15.9 Hz)
                              WR48-63:      41772 (      15.9 Hz)
                           FEE64 disc:    1109863 (     422.9 Hz)
                             MBS info:  203018436 (   77357.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2624.418 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2432.488
  2                0.665                0.000
  3                7.525                0.000
  4                0.000                0.000
  5                0.000             1464.970
  6                0.000                0.000
  7                0.000             1288.051
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        112          0          0          0          0          0          0          0          0          0          0          0
  1    1940572        788          0          0          0          0        394        394          0          0          0          0
  2        442  101676817          0          0         30         30      17294      17294          0  101642169          0          0
  3   40857770  102528388          0          0         55         55      21074      21074    1109863  101376267          0          0
  4        129          0          0          0          0          0          0          0          0          0          0          0
  5    7006930       2878          0          0          0          0       1439       1439          0          0          0          0
  6        548          0          0          0          0          0          0          0          0          0          0          0
  7    7898329       3142          0          0          0          0       1571       1571          0          0          0          0
  8        285          0          0          0          0          0          0          0          0          0          0          0
  9         71          0          0          0          0          0          0          0          0          0          0          0
 10       1135          0          0          0          0          0          0          0          0          0          0          0
 11        122          0          0          0          0          0          0          0          0          0          0          0
 12        679          0          0          0          0          0          0          0          0          0          0          0
 13         92          0          0          0          0          0          0          0          0          0          0          0
 14        666          0          0          0          0          0          0          0          0          0          0          0
 15        106          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.551s ( 1082.882 blocks/s,  67.680 Mb/s)
  358   Sun Jun 6 14:15:05 2021 TDSunday 6 June
14.15 DAQ continues file S496/R32_117
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      System wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     12     12      3      3      4      2      2      1      4      7   : 40572
aida02 :      7      8     10      5      2      3      3      2      1      4      7   : 40604
aida03 :     27     10      4      8      1      4      1      3      2      3      7   : 39740
aida04 :     18      7      4      5      1      3      2      3      2      3      7   : 39712
aida05 :     15     13      8      4      1      3      1      3      1      4      7   : 40548
aida06 :     14      8      8      6      3      2      1      3      1      4      7   : 40568
aida07 :     11      4      5      6      2      3      3      2      1      4      7   : 40540
aida08 :     15      6      5      3      4      3      3      2      1      4      7   : 40604
aida09 :     18      4      4      3      3      2      2      3      1      4      7   : 40648
aida10 :     12      6      8      6      1      1      1      3      1      4      7   : 40288
aida11 :     16      8      6      6      4      4      3      3      2      3      7   : 40352
aida12 :     10      7      4      8      3      4      3      3      2      3      7   : 40288
aida13 :     15      7      5      4      5      4      3      3      2      3      7   : 40324
aida14 :      9      4      6      7      3      4      3      3      2      3      7   : 40260
aida15 :     12      4      8      3      2      2      2      4      1      4      7   : 41136
aida16 :     11      8      6      6      3      4      3      3      2      3      7   : 40268

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_118 - attachment 9
       max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-06_Statistics_aidas-gsi.png
Screenshot_2021-06-06_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 984.png
984.png
Attachment 8: 985.png
985.png
Attachment 9: R32_118
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   48860593 (   17844.1 Hz)
        Other data format:  213059412 (   77810.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        101 (       0.0 Hz)
                               RESUME:        101 (       0.0 Hz)
                              SYNC100:      42200 (      15.4 Hz)
                              WR48-63:      42200 (      15.4 Hz)
                           FEE64 disc:    1112345 (     406.2 Hz)
                             MBS info:  211862465 (   77373.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2738.188 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2613.952
  2                0.575                0.000
  3                7.745                0.000
  4                0.000                0.000
  5                0.000             2154.815
  6                0.000                0.000
  7                0.000             1921.162
  8                0.000              777.447
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        155          0          0          0          0          0          0          0          0          0          0          0
  1      66625         28          0          0          0          0         14         14          0          0          0          0
  2        415  106108157          0          0         26         26      18899      18899          0  106070307          0          0
  3   39298810  106947309          0          0         75         75      21328      21328    1112345  105792158          0          0
  4        171          0          0          0          0          0          0          0          0          0          0          0
  5    4255149       1684          0          0          0          0        842        842          0          0          0          0
  6        596          0          0          0          0          0          0          0          0          0          0          0
  7    5234928       2226          0          0          0          0       1113       1113          0          0          0          0
  8        493          4          0          0          0          0          2          2          0          0          0          0
  9         92          0          0          0          0          0          0          0          0          0          0          0
 10       1212          2          0          0          0          0          1          1          0          0          0          0
 11        160          0          0          0          0          0          0          0          0          0          0          0
 12        781          0          0          0          0          0          0          0          0          0          0          0
 13         89          0          0          0          0          0          0          0          0          0          0          0
 14        724          2          0          0          0          0          1          1          0          0          0          0
 15        193          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.738s ( 1113.497 blocks/s,  69.594 Mb/s)
  357   Sat Jun 5 11:20:15 2021 TDSaturday 5 June
12.25 DAQ continues OK - file S496/R32_48
      alpha background

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     10      6      1      2      4      4      2      2      1      4      7   : 40360
aida02 :      4      3      3      3      3      4      2      2      1      4      7   : 40312
aida03 :     13      5      4      4      2      4      1      3      2      3      7   : 39580
aida04 :     16      5      3      3      2      4      1      3      2      3      7   : 39544
aida05 :      8      6      1      2      3      3      1      3      1      4      7   : 40416
aida06 :      4      3      2      1      3      3      1      3      1      4      7   : 40360
aida07 :      6      6      2      4      4      2      3      2      1      4      7   : 40424
aida08 :      6      4      1      1      4      3      3      2      1      4      7   : 40424
aida09 :     16      1      3      1      0      2      2      3      1      4      7   : 40344
aida10 :      2      5      1      4      1      3      2      4      2      3      7   : 40064
aida11 :     14      3      2      1      4      4      3      3      2      3      7   : 40080
aida12 :     11      2      2      4      4      4      3      3      2      3      7   : 40156
aida13 :     11      3      2      2      3      4      3      3      2      3      7   : 40036
aida14 :      6      5      0      4      3      4      3      3      2      3      7   : 40064
aida15 :     11      6      3      1      3      1      2      4      1      4      7   : 40940
aida16 :     12      5      0      4      4      4      3      3      2      3      7   : 40152

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

      analysis of file S496/R32_47 - attachment 8
       max deadtime 0.35% aida04

13.56 Lost activity monitor - see attachment 9
Attachment 1: Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-06-05_Statistics_aidas-gsi.png
Screenshot_2021-06-05_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 982.png
982.png
Attachment 7: 983.png
983.png
Attachment 8: R32_47
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   47883982 (   17391.8 Hz)
        Other data format:  214036019 (   77739.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        184 (       0.1 Hz)
                               RESUME:        184 (       0.1 Hz)
                              SYNC100:      42255 (      15.3 Hz)
                              WR48-63:      42255 (      15.3 Hz)
                           FEE64 disc:    1053753 (     382.7 Hz)
                             MBS info:  212897388 (   77325.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2753.252 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2696.425
  2                0.825                0.000
  3                9.735                0.000
  4                0.000                0.000
  5                0.000             2175.120
  6                0.000                0.000
  7                0.000             2003.563
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        187          0          0          0          0          0          0          0          0          0          0          0
  1      86597         40          0          0          0          0         20         20          0          0          0          0
  2        381  106659018          0          0         37         37      18746      18746          0  106621452          0          0
  3   38584230  107373193          0          0        147        147      21605      21605    1053753  106275936          0          0
  4        143          0          0          0          0          0          0          0          0          0          0          0
  5    4208337       1756          0          0          0          0        878        878          0          0          0          0
  6        545          0          0          0          0          0          0          0          0          0          0          0
  7    5000124       2012          0          0          0          0       1006       1006          0          0          0          0
  8        478          0          0          0          0          0          0          0          0          0          0          0
  9         96          0          0          0          0          0          0          0          0          0          0          0
 10       1149          0          0          0          0          0          0          0          0          0          0          0
 11        117          0          0          0          0          0          0          0          0          0          0          0
 12        706          0          0          0          0          0          0          0          0          0          0          0
 13        105          0          0          0          0          0          0          0          0          0          0          0
 14        636          0          0          0          0          0          0          0          0          0          0          0
 15        151          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.996s ( 1066.806 blocks/s,  66.675 Mb/s)
Attachment 9: Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png
  356   Fri Jun 4 09:47:44 2021 TDaida15 - system console output - 00.02 Friday 4 June reboot
03/23:58:55|kernel BUG at mm/slab.c:2974!
03/23:58:55|Oops: Exception in kernel mode, sig: 5 [#1]
03/23:58:55|PREEMPT Xilinx Virtex440
03/23:58:55|Modules linked in: aidamem xdriver xh_spidev_register
03/23:58:55|NIP: c009211c LR: c00920b0 CTR: c0038e70
03/23:58:55|REGS: c631fc60 TRAP: 0700   Not tainted  (2.6.31)
03/23:58:55|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
03/23:58:56|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000
03/23:58:56|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 
03/23:58:56|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 
03/23:58:56|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
03/23:58:56|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 
03/23:58:56|NIP [c009211c] cache_alloc_refill+0x130/0x608
03/23:58:56|LR [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:56|Call Trace:
03/23:58:56|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
03/23:58:56|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:56|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:56|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:56|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:56|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:56|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:56|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:56|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:56|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:57|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:57|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:57|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:57|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:57|Instruction dump:
03/23:58:57|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
03/23:58:57|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
03/23:58:57|Kernel panic - not syncing: Fatal exception in interrupt
03/23:58:57|Call Trace:
03/23:58:57|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
03/23:58:57|[c631faf0] [c00345bc] panic+0x94/0x168
03/23:58:57|[c631fb40] [c000bd44] die+0x178/0x18c
03/23:58:57|[c631fb60] [c000c000] _exception+0x164/0x1b4
03/23:58:57|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c
03/23:58:57|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:57|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:57|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:57|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:57|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:58|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:58|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:58|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:58|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:58|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:58|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:58|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:58|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:58|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:58|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
04/00:01:59|
Found 0 errors checking kernel image
04/00:02:00|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
04/00:02:00|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
04/00:02:00|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
04/00:02:00|SPIconfig done now to set up the LMK3200 registers
04/00:02:00|LMK #0 : regInit[0]=0x80000000
04/00:02:00|LMK #0 : regInit[1]=0x10070600
04/00:02:00|LMK #0 : regInit[2]=0x60601
04/00:02:00|LMK #0 : regInit[3]=0x60602
04/00:02:00|LMK #0 : regInit[4]=0x60603
04/00:02:00|LMK #0 : regInit[5]=0x70624
04/00:02:00|LMK #0 : regInit[6]=0x70605
04/00:02:00|LMK #0 : regInit[7]=0x70606
04/00:02:00|LMK #0 : regInit[8]=0x70627
04/00:02:00|LMK #0 : regInit[9]=0x10000908
04/00:02:00|LMK #0 : regInit[10]=0xA0022A09
04/00:02:00|LMK #0 : regInit[11]=0x82800B
04/00:02:01|LMK #0 : regInit[12]=0x28C800D
04/00:02:01|LMK #0 : regInit[13]=0x830020E
04/00:02:01|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
04/00:02:01|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
04/00:02:01|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
04/00:02:01|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
04/00:02:01|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
04/00:02:05|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
04/00:02:11|Using Xilinx Virtex440 machine description
04/00:02:12|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
04/00:02:12|Zone PFN ranges:
04/00:02:12|  DMA      0x00000000 -> 0x00007000
04/00:02:12|  Normal   0x00007000 -> 0x00007000
04/00:02:12|Movable zone start PFN for each node
04/00:02:12|early_node_map[1] active PFN ranges
04/00:02:12|    0: 0x00000000 -> 0x00007000
04/00:02:12|MMU: Allocated 1088 bytes of context maps for 255 contexts
04/00:02:12|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
04/00:02:12|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
04/00:02:12|PID hash table entries: 512 (order: 9, 2048 bytes)
04/00:02:12|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
04/00:02:12|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
04/00:02:12|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
04/00:02:12|Kernel virtual memory layout:
04/00:02:12|  * 0xffffe000..0xfffff000  : fixmap
04/00:02:12|  * 0xfde00000..0xfe000000  : consistent mem
04/00:02:12|  * 0xfde00000..0xfde00000  : early ioremap
04/00:02:13|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
04/00:02:13|NR_IRQS:512
04/00:02:13|clocksource: timebase mult[a00000] shift[22] registered
04/00:02:13|Console: colour dummy device 80x25
04/00:02:13|Mount-cache hash table entries: 512
04/00:02:13|NET: Registered protocol family 16
04/00:02:13|PCI: Probing PCI hardware
04/00:02:13|bio: create slab <bio-0> at 0
04/00:02:13|NET: Registered protocol family 2
04/00:02:13|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
04/00:02:13|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
04/00:02:13|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
04/00:02:13|TCP: Hash tables configured (established 4096 bind 4096)
04/00:02:13|TCP reno registered
04/00:02:13|NET: Registered protocol family 1
04/00:02:13|ROMFS MTD (C) 2007 Red Hat, Inc.
04/00:02:13|msgmni has been set to 214
04/00:02:13|io scheduler noop registered
04/00:02:13|io scheduler anticipatory registered
04/00:02:13|io scheduler deadline registered
04/00:02:13|io scheduler cfq registered (default)
04/00:02:13|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
04/00:02:13|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
04/00:02:14|console [ttyS0] enabled
04/00:02:14|brd: module loaded
04/00:02:14|loop: module loaded
04/00:02:14|Device Tree Probing 'ethernet'
04/00:02:14|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
04/00:02:14|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
04/00:02:14|XLlTemac: DCR address: 0x80
04/00:02:14|XLlTemac: buffer descriptor size: 32768 (0x8000)
04/00:02:14|XLlTemac: Allocating DMA descriptors with kmalloc
04/00:02:14|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
04/00:02:14|XTemac: PHY detected at address 7.
04/00:02:14|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
04/00:02:14|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14|Using buffer write method
04/00:02:15|cfi_cmdset_0001: Erase suspend on write enabled
04/00:02:15|cmdlinepart partition parsing not available
04/00:02:15|RedBoot partition parsing not available
04/00:02:15|Creating 5 MTD partitions on "fc000000.flash":
04/00:02:15|0x000000000000-0x000000500000 : "golden_firmware"
04/00:02:15|0x000000500000-0x000000800000 : "golden_kernel"
04/00:02:15|0x000000800000-0x000000d00000 : "user_firmware"
04/00:02:15|0x000000d00000-0x000000fe0000 : "user_kernel"
04/00:02:15|0x000000fe0000-0x000001000000 : "env_variables"
04/00:02:15|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
04/00:02:15|SPI: XIlinx spi: bus number now 32766
04/00:02:15|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
04/00:02:15|SPI: XIlinx spi: bus number now 32765
04/00:02:15|mice: PS/2 mouse device common for all mice
04/00:02:15|Device Tree Probing 'i2c'
04/00:02:15| #0 at 0x81600000 mapped to 0xD1030000, irq=22
04/00:02:15|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
04/00:02:15|TCP cubic registered
04/00:02:15|NET: Registered protocol family 17
04/00:02:15|RPC: Registered udp transport module.
04/00:02:15|RPC: Registered tcp transport module.
04/00:02:16|eth0: XLlTemac: Options: 0x3fa
04/00:02:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
04/00:02:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
04/00:02:16|eth0: XLlTemac: speed set to 1000Mb/s
04/00:02:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
04/00:02:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
04/00:02:18|Sending DHCP requests ., OK
04/00:02:19|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15
04/00:02:20|IP-Config: Complete:
04/00:02:20|     device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255,
04/00:02:20|     host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics,
04/00:02:20|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15
04/00:02:20|Looking up port of RPC 100003/2 on 192.168.11.99
04/00:02:20|Looking up port of RPC 100005/1 on 192.168.11.99
04/00:02:20|VFS: Mounted root (nfs filesystem) on device 0:12.
04/00:02:20|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
04/00:02:20|Starting sysinit...
04/00:02:21|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
04/00:02:22|		Press 'I' to enter interactive startup.
04/00:02:22|Setting clock  (utc): Thu Jun  3 23:02:22 BST 2021 [  OK  ]
04/00:02:22|Building the cache [  OK  ]
04/00:02:22|Setting hostname aida15:  [  OK  ]
04/00:02:23|Mounting local filesystems:  [  OK  ]
04/00:02:25|Enabling /etc/fstab swaps:  [  OK  ]
04/00:02:27|Finishing sysinit...
INIT: Entering runlevel: 3
04/00:02:30|Entering non-interactive startup
04/00:02:30|FATAL: Module ipv6 not found.
04/00:02:31|Bringing up loopback interface:  [  OK  ]
04/00:02:33|FATAL: Module ipv6 not found.
04/00:02:34|Starting system logger: [  OK  ]
04/00:02:34|Starting kernel logger: [  OK  ]
04/00:02:35|Starting rpcbind: [  OK  ]
04/00:02:36|Mounting NFS filesystems:  [  OK  ]
04/00:02:37|Mounting other filesystems:  [  OK  ]
04/00:02:37|Starting xinetd: [  OK  ]
04/00:02:38|Starting midas:  Starting MIDAS Data Acquisition for aida15
04/00:02:38|xaida: device parameters: base=0x81000000 size=0x200000
04/00:02:44|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
04/00:02:44|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
04/00:02:44|xaida: driver assigned major number 254
04/00:02:44|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
04/00:02:50|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
04/00:02:50|AIDAMEM: aidamem: driver assigned major number 253
04/00:02:50|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
04/00:02:56|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
04/00:02:57|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
04/00:02:57|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
04/00:02:57|Computer Name = aida15; Temp Directory = /tmp/tcl361
04/00:03:01|package limit is not available: can't find package limit
04/00:03:04|Running with default file descriptor limit
04/00:03:04|package setuid is not available: can't find package setuid
04/00:03:05|Running as user 0 group 0
04/00:03:05|
04/00:03:06|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
04/00:03:06|
04/00:03:06|Built without pthreads
04/00:03:06|
04/00:03:06|Creating NetVars
04/00:03:06|Output buffer length = 65xaida: open:
04/00:03:06|504; format option = 4AIDAMEM: aidamem_open:
04/00:03:06|; transfer option = 3
04/00:03:06|EB transfer option = 3
04/00:03:06|NetVars created and initialised
04/00:03:06|Statistics thread starting
04/00:03:06|Statistics thread created
04/00:03:06|Stat/Rate creation thread starting
04/00:03:06|Data Acquisition task has PID 375
04/00:03:06|Stat/Rate creation thread created
04/00:03:06|Hit/Rate creation thread starting
04/00:03:06|Hit/Rate creation thread created
04/00:03:07|AIDA Heartbeat thread starting
04/00:03:07|Heartbeat thread created
04/00:03:07|Installing signal handlers
04/00:03:07|Done
04/00:03:07|ModuleNum = 0
04/00:03:07|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706 
04/00:03:07|Spectra table initialised
04/00:03:07|AIDA Data Acquisition now all ready to start
04/00:03:07|SIGBUS, SIGSEGV and SIGPIPE traps setup
04/00:03:07|/debug user "debug" password " bjkhcfyuqis"
04/00:03:14|httpd started on port 8015
04/00:03:14|
04/00:03:14|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl
04/00:03:14|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl
04/00:03:14|XAIDA Access package 1.0
04/00:03:16|/XAIDAAccessServer
04/00:03:16|XAD9252 Access package 1.0
04/00:03:16|/XAD9252AccessServer
04/00:03:16|/DataBaseAccessServer
04/00:03:16|[  OK  ]
04/00:03:16|/NetVarService
04/00:03:17|/SigTaskService
04/00:03:17|Loaded MemSasAccess
04/00:03:17|/SpectrumService
04/00:03:17|loading tcl/AIDARunControl.tcl for namespace ::
04/00:03:17|
04/00:03:17|DENX ELDK version 4.2 build 2008-04-01
04/00:03:17|Linux 2.6.31 on a ppc
04/00:03:17|
04/00:03:17|aida15 login: /DataAcquisitionControlServer
04/00:03:17|DefineMessage unknown
04/00:03:17|Run Control Server Implementation for AIDA
04/00:03:17|RunControlServer loaded
04/00:03:17|loading Html/RunControl/implementation.tcl
04/00:03:18|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
04/00:03:18|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
04/00:03:18|Created UI registers
04/00:03:18|RunControl loaded
04/00:03:19|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
04/00:03:19|AIDA RunControl loaded
04/00:03:19|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
04/00:03:19|do_GetState returned z=0 and 8
04/09:31:05|do_GetState returned z=0 and 8
04/09:31:17|do_GetState returned z=0 and 8
04/09:31:30|do_GetState returned z=0 and 8
04/09:31:43|do_GetState returned z=0 and 8
04/09:31:55|do_GetState returned z=0 and 8
04/09:32:07|do_GetState returned z=0 and 8
04/09:32:19|do_GetState returned z=0 and 8
04/09:32:32|do_GetState returned z=0 and 8
04/09:32:45|do_GetState returned z=0 and 8
04/09:32:57|do_GetState returned z=0 and 8
04/09:33:10|do_GetState returned z=0 and 8
04/09:33:22|do_GetState returned z=0 and 8
04/09:33:35|do_GetState returned z=0 and 8
04/09:33:47|do_GetState returned z=0 and 8
04/09:34:00|do_GetState returned z=0 and 8
04/09:34:13|do_GetState returned z=0 and 8
04/09:34:25|do_GetState returned z=0 and 8
04/09:34:38|do_GetState returned z=0 and 8
04/09:34:50|do_GetState returned z=0 and 8
04/09:35:03|do_GetState returned z=0 and 8
04/09:35:16|do_GetState returned z=0 and 8
04/09:35:28|do_GetState returned z=0 and 8
04/09:35:41|do_GetState returned z=0 and 8
04/09:35:53|do_GetState returned z=0 and 8
04/09:36:06|do_GetState returned z=0 and 8
04/09:36:18|do_GetState returned z=0 and 8
04/09:36:31|do_GetState returned z=0 and 8
04/09:36:43|do_GetState returned z=0 and 8
04/09:36:56|do_GetState returned z=0 and 8
04/09:37:09|do_GetState returned z=0 and 8
04/09:37:21|do_GetState returned z=0 and 8
04/09:37:34|do_GetState returned z=0 and 8
04/09:37:46|do_GetState returned z=0 and 8
04/09:37:59|do_GetState returned z=0 and 8
04/09:38:12|do_GetState returned z=0 and 8
04/09:38:24|do_GetState returned z=0 and 8
04/09:38:37|do_GetState returned z=0 and 8
04/09:38:49|do_GetState returned z=0 and 8
04/09:39:02|do_GetState returned z=0 and 8
04/09:39:15|do_GetState returned z=0 and 8
04/09:39:27|do_GetState returned z=0 and 8
04/09:39:40|do_GetState returned z=0 and 8
04/09:39:52|do_GetState returned z=0 and 8
04/09:40:05|do_GetState returned z=0 and 8
04/09:40:18|do_GetState returned z=0 and 8
04/09:40:30|do_GetState returned z=0 and 8
04/09:40:43|do_GetState returned z=0 and 8
04/09:40:55|do_GetState returned z=0 and 8
04/09:41:08|do_GetState returned z=0 and 8
04/09:41:21|do_GetState returned z=0 and 8
04/09:41:33|do_GetState returned z=0 and 8
04/09:41:46|do_GetState returned z=0 and 8
04/09:41:58|do_GetState returned z=0 and 8
04/09:42:11|do_GetState returned z=0 and 8
04/09:42:23|do_GetState returned z=0 and 8
04/09:42:36|do_GetState returned z=0 and 8
04/09:42:48|do_GetState returned z=0 and 8
04/09:43:01|do_GetState returned z=0 and 8
04/09:43:14|do_GetState returned z=0 and 8
04/09:43:26|do_GetState returned z=0 and 8
04/09:43:39|do_GetState returned z=0 and 8
04/09:43:51|do_GetState returned z=0 and 8
04/09:44:04|do_GetState returned z=0 and 8
04/09:44:17|do_GetState returned z=0 and 8
04/09:44:29|do_GetState returned z=0 and 8
04/09:44:42|do_GetState returned z=0 and 8
04/09:44:54|do_GetState returned z=0 and 8
04/09:45:07|do_GetState returned z=0 and 8
04/09:45:19|do_GetState returned z=0 and 8
04/09:45:32|do_GetState returned z=0 and 8
04/09:45:44|do_GetState returned z=0 and 8
04/09:45:57|do_GetState returned z=0 and 8
04/09:46:09|do_GetState returned z=0 and 8
04/09:46:22|do_GetState returned z=0 and 8
04/09:46:34|do_GetState returned z=0 and 8
04/09:46:47|do_GetState returned z=0 and 8
04/09:47:00|do_GetState returned z=0 and 8
04/09:47:12|do_GetState returned z=0 and 8
04/09:47:25|do_GetState returned z=0 and 8
04/09:47:37|do_GetState returned z=0 and 8
04/09:47:50|do_GetState returned z=0 and 8
04/09:48:03|do_GetState returned z=0 and 8
04/09:48:15|do_GetState returned z=0 and 8
04/09:48:28|do_GetState returned z=0 and 8
04/09:48:40|do_GetState returned z=0 and 8
04/09:48:53|do_GetState returned z=0 and 8
04/09:49:05|do_GetState returned z=0 and 8
04/09:49:18|do_GetState returned z=0 and 8
04/09:49:31|do_GetState returned z=0 and 8
04/09:49:43|do_GetState returned z=0 and 8
04/09:49:56|do_GetState returned z=0 and 8
04/09:50:08|do_GetState returned z=0 and 8
04/09:50:21|do_GetState returned z=0 and 8
04/09:50:33|do_GetState returned z=0 and 8
04/09:50:46|do_GetState returned z=0 and 8
04/09:50:58|do_GetState returned z=0 and 8
04/09:51:11|do_GetState returned z=0 and 8
04/09:51:23|do_GetState returned z=0 and 8
04/09:51:36|do_GetState returned z=0 and 8
04/09:51:49|do_GetState returned z=0 and 8
04/09:52:01|do_GetState returned z=0 and 8
04/09:52:14|do_GetState returned z=0 and 8
04/09:52:26|do_GetState returned z=0 and 8
04/09:52:39|do_GetState returned z=0 and 8
04/09:52:51|do_GetState returned z=0 and 8
04/09:53:04|do_GetState returned z=0 and 8
04/09:53:16|do_GetState returned z=0 and 8
04/09:53:29|do_GetState returned z=0 and 8
04/09:53:42|do_GetState returned z=0 and 8
04/09:53:54|do_GetState returned z=0 and 8
04/09:54:07|do_GetState returned z=0 and 8
04/09:54:19|do_GetState returned z=0 and 8
04/09:54:32|do_GetState returned z=0 and 8
04/09:54:44|do_GetState returned z=0 and 8
04/09:54:57|do_GetState returned z=0 and 8
04/09:55:09|do_GetState returned z=0 and 8
04/09:55:22|do_GetState returned z=0 and 8
04/09:55:34|do_GetState returned z=0 and 8
04/09:55:47|do_GetState returned z=0 and 8
04/09:56:00|do_GetState returned z=0 and 8
04/09:56:12|do_GetState returned z=0 and 8
04/09:56:25|do_GetState returned z=0 and 8
04/09:56:37|do_GetState returned z=0 and 8
04/09:56:50|do_GetState returned z=0 and 8
04/09:57:02|do_GetState returned z=0 and 8
04/09:57:15|do_GetState returned z=0 and 8
04/09:57:27|do_GetState returned z=0 and 8
04/09:57:40|do_GetState returned z=0 and 8
04/09:57:53|do_GetState returned z=0 and 8
04/09:58:05|do_GetState returned z=0 and 8
04/09:58:18|do_GetState returned z=0 and 8
04/09:58:30|do_GetState returned z=0 and 8
04/09:58:43|do_GetState returned z=0 and 8
04/09:58:56|do_GetState returned z=0 and 8
04/09:59:08|do_GetState returned z=0 and 8
04/09:59:21|do_GetState returned z=0 and 8
04/09:59:33|do_GetState returned z=0 and 8
04/09:59:46|do_GetState returned z=0 and 8
04/09:59:58|do_GetState returned z=0 and 8
04/10:00:11|do_GetState returned z=0 and 8
04/10:00:23|do_GetState returned z=0 and 8
04/10:00:36|do_GetState returned z=0 and 8
04/10:00:48|do_GetState returned z=0 and 8
04/10:01:01|do_GetState returned z=0 and 8
04/10:01:13|do_GetState returned z=0 and 8
04/10:01:26|do_GetState returned z=0 and 8
04/10:01:38|do_GetState returned z=0 and 8
04/10:01:51|do_GetState returned z=0 and 8
04/10:02:04|do_GetState returned z=0 and 8
04/10:02:16|do_GetState returned z=0 and 8
04/10:02:29|do_GetState returned z=0 and 8
04/10:02:41|do_GetState returned z=0 and 8
04/10:02:54|do_GetState returned z=0 and 8
04/10:03:06|do_GetState returned z=0 and 8
04/10:03:19|do_GetState returned z=0 and 8
04/10:03:31|do_GetState returned z=0 and 8
04/10:03:44|do_GetState returned z=0 and 8
04/10:03:57|do_GetState returned z=0 and 8
04/10:04:09|do_GetState returned z=0 and 8
04/10:04:22|do_GetState returned z=0 and 8
04/10:04:34|do_GetState returned z=0 and 8
04/10:04:47|do_GetState returned z=0 and 8
04/10:04:59|do_GetState returned z=0 and 8
04/10:05:12|do_GetState returned z=0 and 8
04/10:05:25|do_GetState returned z=0 and 8
04/10:05:37|do_GetState returned z=0 and 8
04/10:05:50|do_GetState returned z=0 and 8
04/10:06:02|do_GetState returned z=0 and 8
04/10:06:15|do_GetState returned z=0 and 8
04/10:06:27|do_GetState returned z=0 and 8
04/10:06:40|do_GetState returned z=0 and 8
04/10:06:52|do_GetState returned z=0 and 8
04/10:07:05|do_GetState returned z=0 and 8
04/10:07:18|do_GetState returned z=0 and 8
04/10:07:30|do_GetState returned z=0 and 8
  355   Fri Jun 4 08:40:52 2021 TDFriday 4 June
09.20 DAQ found stopped
      files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight
      aida15 rebooted at 00.02

      regained control of DAQ and issued DAQ STOP by restarting NewMerger *twice*

11.00 DAQ reset and setup
      All system wide checks OK *except*

		 Base 		Current 	Difference
aida05 fault 	 0x4d3c : 	 0x4d3c : 	 0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      4      2      8      5      4      1      2      2      4      7   : 41352
aida02 :      0      3      3      6      6      4      1      2      2      4      7   : 41352
aida03 :      2      4      2     12      1      3      2      2      2      4      7   : 41352
aida04 :      2      2      1      7      4      3      2      2      2      4      7   : 41352
aida05 :      2      3      1      5      3      2      3      2      2      4      7   : 41360
aida06 :      2      4      2      6      4      3      2      2      2      4      7   : 41352
aida07 :      1      5      2      9      4      2      4      3      1      4      7   : 41324
aida08 :      0      3      1      7      4      3      4      3      1      4      7   : 41352
aida09 :      3      3      0      8      2      2      3      2      2      4      7   : 41380
aida10 :      1      2      2     11      3      2      3      3      3      3      7   : 41044
aida11 :      2      3      1      6      6      4      2      3      3      3      7   : 41072
aida12 :      4      2      1      8      3      3      3      3      1      4      7   : 41072
aida13 :      2      4      1      8      4      4      2      3      3      3      7   : 41016
aida14 :      1      4      1      9      4      4      2      3      3      3      7   : 41044
aida15 :      0      5      1      2      2      1      2      4      3      4      7   : 42872
aida16 :      1      3      2      9      4      4      4      4      2      3      7   : 41052

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

11.03 DAQ resumes file S496/R32
      alpha background

      slow comparator 0x64
      BNC PB-5 pulser OFF

11.09 1.8.W spectra - 20us FSR - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

13.07 analysis of file S496/R32_16 - attachment 8
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 2: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-04_Statistics_aidas-gsi.png
Screenshot_2021-06-04_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 980.png
980.png
Attachment 7: 981.png
981.png
Attachment 8: R32_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   49166189 (   17465.0 Hz)
        Other data format:  212753815 (   75575.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        137 (       0.0 Hz)
                               RESUME:        137 (       0.0 Hz)
                              SYNC100:      42486 (      15.1 Hz)
                              WR48-63:      42486 (      15.1 Hz)
                           FEE64 disc:    1037793 (     368.6 Hz)
                             MBS info:  211630776 (   75176.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2815.130 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2759.088
  2                0.741                0.000
  3                4.946                0.000
  4                0.000                0.000
  5                0.000             2173.993
  6                0.000                0.000
  7                0.000             2104.432
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        184          0          0          0          0          0          0          0          0          0          0          0
  1     115458         62          0          0          0          0         31         31          0          0          0          0
  2        444  105932150          0          0         36         36      18754      18754          0  105894570          0          0
  3   39683483  106817705          0          0        101        101      21752      21752    1037793  105736206          0          0
  4        154          0          0          0          0          0          0          0          0          0          0          0
  5    4413391       1852          0          0          0          0        926        926          0          0          0          0
  6        586          0          0          0          0          0          0          0          0          0          0          0
  7    4948958       2044          0          0          0          0       1022       1022          0          0          0          0
  8        489          0          0          0          0          0          0          0          0          0          0          0
  9         77          0          0          0          0          0          0          0          0          0          0          0
 10       1172          0          0          0          0          0          0          0          0          0          0          0
 11        163          2          0          0          0          0          1          1          0          0          0          0
 12        754          0          0          0          0          0          0          0          0          0          0          0
 13         94          0          0          0          0          0          0          0          0          0          0          0
 14        605          0          0          0          0          0          0          0          0          0          0          0
 15        177          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.027s ( 1102.409 blocks/s,  68.901 Mb/s)
  354   Thu Jun 3 10:16:39 2021 TDThursday 3 June
11.19 Hot HEC channels multiple FEE64s
      ASIC check fixes

      System wide checks OK * except*

		 Base 		Current 	Difference
aida01 fault 	 0x8940 : 	 0x8941 : 	 1  
aida02 fault 	 0x9b32 : 	 0x9b33 : 	 1  
aida03 fault 	 0xf266 : 	 0xf267 : 	 1  
aida04 fault 	 0x579b : 	 0x579c : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      4      2      2      2      2      2      3      2      4      7   : 41516
aida02 :     10      7      2      0      1      2      2      3      2      4      7   : 41408
aida03 :     10      8      2      2      2      4      1      4      2      3      7   : 40008
aida04 :      5      4      4      2      2      3      1      4      2      3      7   : 39860
aida05 :     12      3      2      2      1      2      2      3      2      4      7   : 41448
aida06 :     15      9      2      2      3      2      1      3      2      4      7   : 41380
aida07 :     12      8      1      2      2      3      3      2      2      4      7   : 41408
aida08 :      9      6      3      1      2      3      3      2      2      4      7   : 41380
aida09 :     15      6      1      1      2      2      2      3      2      4      7   : 41500
aida10 :     15      6      3      2      1      3      3      2      1      4      7   : 40348
aida11 :      8      2      1      1      1      3      1      3      2      4      7   : 41248
aida12 :     18      3      2      1      3      3      2      2      2      4      7   : 41184
aida13 :      7      4      1      4      3      4      3      3      1      4      7   : 41100
aida14 :     18      6      1      4      3      4      2      2      1      4      7   : 40392
aida15 :      7      7      1      3      1      2      1      3      2      4      7   : 41220
aida16 :      4      4      2      3      3      3      2      4      1      4      7   : 41200

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Jun 02 16:27:30 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36

13.09 analysis of file S496/R31_16 - attachment 8
       max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-06-03_Statistics_aidas-gsi.png
Screenshot_2021-06-03_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 970.png
970.png
Attachment 7: 971.png
971.png
Attachment 8: R31_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   50135749 (   17884.4 Hz)
        Other data format:  211784253 (   75547.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.1 Hz)
                               RESUME:        199 (       0.1 Hz)
                              SYNC100:      42443 (      15.1 Hz)
                              WR48-63:      42443 (      15.1 Hz)
                           FEE64 disc:    1081490 (     385.8 Hz)
                             MBS info:  210617479 (   75131.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          4 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2803.320 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2522.967
  2                0.535                0.000
  3                8.636                0.000
  4                0.000                0.000
  5                0.000             2119.634
  6                0.000                0.000
  7                0.000             2067.036
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        196          2          0          0          0          0          0          0          2          0          0          2
  1     193203         74          0          0          0          0         37         37          0          0          0          0
  2        471  105497692          0          0         26         26      18605      18605          0  105460430          0          0
  3   40159698  106282506          0          0        173        173      21812      21812    1081487  105157049          0          0
  4        128          0          0          0          0          0          0          0          0          0          0          0
  5    4517663       1914          0          0          0          0        957        957          0          0          0          0
  6        641          2          0          0          0          0          1          1          0          0          0          0
  7    5260254       2060          0          0          0          0       1030       1030          0          0          0          0
  8        463          0          0          0          0          0          0          0          0          0          0          0
  9        104          0          0          0          0          0          0          0          0          0          0          0
 10       1026          2          0          0          0          0          1          1          0          0          0          0
 11        141          1          0          0          0          0          0          0          1          0          0          1
 12        786          0          0          0          0          0          0          0          0          0          0          0
 13        109          0          0          0          0          0          0          0          0          0          0          0
 14        687          0          0          0          0          0          0          0          0          0          0          0
 15        179          0          0          0          0          0          0          0          0          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.445s ( 1124.966 blocks/s,  70.310 Mb/s)
  353   Wed Jun 2 14:53:05 2021 OH, TDMerger/Tapeserver issues after reboot of aida-3
On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen.
We were also unable to connected to MIDAS via ssh port forwarding.

Since the restart we have been unable to write to file.

The tapeserver will allocate/mount/open a file but nothing will be written to it.
The merger will be receiving data but never shows as having any "Current links with data".

/MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23  2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119

The current disk structure is:
[npg@aidas-gsi npg]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
sdb      8:16   0   7.3T  0 disk 
&#9500;&#9472;sdb1   8:17   0   200M  0 part 
&#9500;&#9472;sdb2   8:18   0   500M  0 part 
&#9492;&#9472;sdb3   8:19   0   7.3T  0 part 
  &#9500;&#9472;vg_aidas2-lv_root (dm-2)
       253:2    0    50G  0 lvm  
  &#9500;&#9472;vg_aidas2-lv_home (dm-3)
       253:3    0   7.2T  0 lvm  /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
  &#9492;&#9472;vg_aidas2-lv_swap (dm-4)
       253:4    0   7.8G  0 lvm  
sdc      8:32   0   7.3T  0 disk 
&#9492;&#9472;sdc1   8:33   0   7.3T  0 part /media/ThirdDrive
sdd      8:48   0   7.3T  0 disk 
&#9492;&#9472;sdd1   8:49   0   7.3T  0 part /media/SecondDrive
sda      8:0    0 465.8G  0 disk 
&#9500;&#9472;sda1   8:1    0   200M  0 part /boot/efi
&#9500;&#9472;sda2   8:2    0   500M  0 part /boot
&#9492;&#9472;sda3   8:3    0 465.1G  0 part 
  &#9500;&#9472;vg_aidasgsi-lv_root (dm-0)
       253:0    0    50G  0 lvm  /
  &#9500;&#9472;vg_aidasgsi-lv_swap (dm-1)
       253:1    0   7.8G  0 lvm  [SWAP]
  &#9492;&#9472;vg_aidasgsi-lv_home (dm-5)
       253:5    0 407.3G  0 lvm  /home


Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3

TapeServer config file - attachment 4
Merger options - attachment 5

Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png
Attachment 1: 210602_MergerTape_HTTPd_Startup
tidy up
tclsh8.5_copy2: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl25624
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "z5y30nlwy2e4"
httpd started on port 8115

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
/TapeServer
loading tcl/NewMergerControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl
/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
RunControl loaded
loading Html/NewMerger/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
NewMerge Control loaded
Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl
Shared memory area located at 0x7f34d3028000
Tape Server comms table located at 0x7f34d3028000
MERGE command  30
setup MERGER
Stop Merge
Halt (1)
Action has completed: MergeCommand 2 Halt
Setup Merge
Setup (1)
Action has completed: MergeCommand 1 Setup
MERGE command  32
go MERGER
Go Merge
Go (1)
Action has completed: MergeCommand 3 Go
link 0 GO (1)
Action has completed: MergeCommand 22 {link 0 GO} 0 2
link 1 GO (1)
Action has completed: MergeCommand 22 {link 1 GO} 1 2
link 2 GO (1)
Action has completed: MergeCommand 22 {link 2 GO} 2 2
link 3 GO (1)
Action has completed: MergeCommand 22 {link 3 GO} 3 2
link 4 GO (1)
Action has completed: MergeCommand 22 {link 4 GO} 4 2
link 5 GO (1)
Action has completed: MergeCommand 22 {link 5 GO} 5 2
link 6 GO (1)
Action has completed: MergeCommand 22 {link 6 GO} 6 2
link 7 GO (1)
Action has completed: MergeCommand 22 {link 7 GO} 7 2
link 8 GO (1)
Action has completed: MergeCommand 22 {link 8 GO} 8 2
link 9 GO (1)
Action has completed: MergeCommand 22 {link 9 GO} 9 2
link 10 GO (1)
Action has completed: MergeCommand 22 {link 10 GO} 10 2
link 11 GO (1)
Action has completed: MergeCommand 22 {link 11 GO} 11 2
link 12 GO (1)
Action has completed: MergeCommand 22 {link 12 GO} 12 2
link 13 GO (1)
Action has completed: MergeCommand 22 {link 13 GO} 13 2
link 14 GO (1)
Action has completed: MergeCommand 22 {link 14 GO} 14 2
link 15 GO (1)
Action has completed: MergeCommand 22 {link 15 GO} 15 2
Resume MERGER

Attachment 2: 210602_NewMerger_Startup
Tidy up
MERGE: no process killed
merge64.AD: no process killed
Starting New Merger

[1] 25659
MERGE Master (25659): Message logger not contacted.
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
Creating NetVars
Creating NetVars complete
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
MERGE Master (25659): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0
MERGE Master (25659): File mapped object /SHM_11000 of size 536628 created
MERGE Master (25659): Memory mapping 536628 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a99b8000.
MERGE Master (25659): File mapped object /SHM_11001 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a39b6000.
MERGE Master (25659): File mapped object /SHM_11002 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc09d9b4000.
MERGE Master (25659): File mapped object /SHM_11003 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0979b2000.
MERGE Master (25659): File mapped object /SHM_11004 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0919b0000.
MERGE Master (25659): File mapped object /SHM_11005 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc08b9ae000.
MERGE Master (25659): File mapped object /SHM_11006 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0859ac000.
MERGE Master (25659): File mapped object /SHM_11007 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc07f9aa000.
MERGE Master (25659): File mapped object /SHM_11008 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0799a8000.
MERGE Master (25659): File mapped object /SHM_11009 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0739a6000.
MERGE Master (25659): File mapped object /SHM_11010 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc06d9a4000.
MERGE Master (25659): File mapped object /SHM_11011 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0679a2000.
MERGE Master (25659): File mapped object /SHM_11012 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0619a0000.
MERGE Master (25659): File mapped object /SHM_11013 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05b99e000.
MERGE Master (25659): File mapped object /SHM_11014 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05599c000.
MERGE Master (25659): File mapped object /SHM_11015 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc04f99a000.
MERGE Master (25659): File mapped object /SHM_11016 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Starting stats/rates task
MERGE Master (25659): Stats/Rates task has pid 25661
MERGE Master (25659): Starting 16 link tasks
MERGE Master (25659): Link task has pid 25662
MERGE Master (25659): Link task has pid 25663
MERGE Statistics/Rates (25661): Message logger not contacted.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
Creating NetVars
Creating NetVars complete
MERGE Master (25659): Link task has pid 25664
MERGE Data Link (25662): Message logger not contacted.
MERGE Data Link (25662): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25662): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0
MERGE Data Link (25662): File mapped object /SHM_11000 accessed
MERGE Data Link (25662): Memory mapping 536628 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f198ee7a000.
MERGE Data Link (25662): Allocated 71720 bytes for local input buffer
MERGE Data Link (25662): File mapped object /SHM_11001 accessed
MERGE Data Link (25662): Memory mapping 100669480 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Master (25659): Link task has pid 25665
MERGE Data Link (25663): Message logger not contacted.
MERGE Data Link (25663): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25663): Configuration: index = 1, SHM key=11000, TCP port base = 11002, module = 0, hardware = 0
MERGE Data Link (25663): File mapped object /SHM_11000 accessed
MERGE Data Link (25663): Memory mapping 536628 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3bee603000.
MERGE Data Link (25663): Allocated 71720 bytes for local input buffer
MERGE Data Link (25663): File mapped object /SHM_11002 accessed
MERGE Data Link (25663): Memory mapping 100669480 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Master (25659): Link task has pid 25666
MERGE Data Link (25664): Message logger not contacted.
MERGE Data Link (25665): Message logger not contacted.
MERGE Data Link (25664): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (25665): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
Creating NetVars complete
MERGE Data Link (25665): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0
MERGE Data Link (25665): File mapped object /SHM_11000 accessed
MERGE Data Link (25665): Memory mapping 536628 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7418ae6000.
MERGE Data Link (25665): Allocated 71720 bytes for local input buffer
MERGE Data Link (25665): File mapped object /SHM_11004 accessed
MERGE Data Link (25665): Memory mapping 100669480 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
Creating NetVars complete
MERGE Data Link (25664): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0
MERGE Data Link (25664): File mapped object /SHM_11000 accessed
MERGE Data Link (25664): Memory mapping 536628 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da72c8000.
MERGE Data Link (25664): Allocated 71720 bytes for local input buffer
MERGE Data Link (25664): File mapped object /SHM_11003 accessed
MERGE Data Link (25664): Memory mapping 100669480 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Master (25659): Link task has pid 25667
MERGE Master (25659): Link task has pid 25668
MERGE Data Link (25666): Message logger not contacted.
MERGE Data Link (25666): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25666): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0
MERGE Data Link (25666): File mapped object /SHM_11000 accessed
MERGE Data Link (25666): Memory mapping 536628 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7f4ec2000.
MERGE Data Link (25666): Allocated 71720 bytes for local input buffer
MERGE Data Link (25666): File mapped object /SHM_11005 accessed
MERGE Data Link (25666): Memory mapping 100669480 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Master (25659): Link task has pid 25669
MERGE Data Link (25667): Message logger not contacted.
MERGE Data Link (25667): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25667): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0
MERGE Data Link (25667): File mapped object /SHM_11000 accessed
MERGE Data Link (25667): Memory mapping 536628 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b878c9000.
MERGE Data Link (25667): Allocated 71720 bytes for local input buffer
MERGE Data Link (25667): File mapped object /SHM_11006 accessed
MERGE Data Link (25667): Memory mapping 100669480 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Master (25659): Link task has pid 25670
MERGE Data Link (25668): Message logger not contacted.
MERGE Data Link (25668): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25671
Creating NetVars complete
MERGE Data Link (25668): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0
MERGE Data Link (25668): File mapped object /SHM_11000 accessed
MERGE Data Link (25668): Memory mapping 536628 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c7c9a000.
MERGE Data Link (25668): Allocated 71720 bytes for local input buffer
MERGE Data Link (25668): File mapped object /SHM_11007 accessed
MERGE Data Link (25668): Memory mapping 100669480 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25669): Message logger not contacted.
MERGE Data Link (25669): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25672
Creating NetVars complete
MERGE Data Link (25669): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0
MERGE Data Link (25669): File mapped object /SHM_11000 accessed
MERGE Data Link (25669): Memory mapping 536628 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e7704000.
MERGE Data Link (25669): Allocated 71720 bytes for local input buffer
MERGE Data Link (25669): File mapped object /SHM_11008 accessed
MERGE Data Link (25669): Memory mapping 100669480 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e1702000.
MERGE Data Link (25671): Message logger not contacted.
MERGE Data Link (25671): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25671): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0
MERGE Data Link (25671): File mapped object /SHM_11000 accessed
MERGE Data Link (25671): Memory mapping 536628 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38b4f59000.
MERGE Data Link (25671): Allocated 71720 bytes for local input buffer
MERGE Data Link (25671): File mapped object /SHM_11010 accessed
MERGE Data Link (25671): Memory mapping 100669480 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38aef57000.
MERGE Data Link (25670): Message logger not contacted.
MERGE Data Link (25670): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25673
Creating NetVars complete
MERGE Data Link (25670): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0
MERGE Data Link (25670): File mapped object /SHM_11000 accessed
MERGE Data Link (25670): Memory mapping 536628 bytes
MERGE Data Link (25672): Message logger not contacted.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff7744ce000.
MERGE Data Link (25672): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (25670): Allocated 71720 bytes for local input buffer
MERGE Data Link (25670): File mapped object /SHM_11009 accessed
MERGE Data Link (25670): Memory mapping 100669480 bytes
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
Creating NetVars complete
MERGE Data Link (25672): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0
MERGE Data Link (25672): File mapped object /SHM_11000 accessed
MERGE Data Link (25672): Memory mapping 536628 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bb287e000.
MERGE Data Link (25672): Allocated 71720 bytes for local input buffer
MERGE Data Link (25672): File mapped object /SHM_11011 accessed
MERGE Data Link (25672): Memory mapping 100669480 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
MERGE Master (25659): Link task has pid 25674
MERGE Data Link (25673): Message logger not contacted.
MERGE Data Link (25673): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25673): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0
MERGE Data Link (25673): File mapped object /SHM_11000 accessed
MERGE Data Link (25673): Memory mapping 536628 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3cd54a000.
MERGE Data Link (25673): Allocated 71720 bytes for local input buffer
MERGE Data Link (25673): File mapped object /SHM_11012 accessed
MERGE Data Link (25673): Memory mapping 100669480 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Master (25659): Link task has pid 25675
MERGE Data Link (25674): Message logger not contacted.
MERGE Data Link (25674): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25674): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0
MERGE Data Link (25674): File mapped object /SHM_11000 accessed
MERGE Data Link (25674): Memory mapping 536628 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faec5dbd000.
MERGE Data Link (25674): Allocated 71720 bytes for local input buffer
MERGE Data Link (25674): File mapped object /SHM_11013 accessed
MERGE Data Link (25674): Memory mapping 100669480 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faebfdbb000.
MERGE Master (25659): Link task has pid 25676
MERGE Master (25659): Link task has pid 25677
MERGE Data Link (25675): Message logger not contacted.
MERGE Data Link (25675): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25675): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0
MERGE Data Link (25675): File mapped object /SHM_11000 accessed
MERGE Data Link (25675): Memory mapping 536628 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef43120000.
MERGE Data Link (25675): Allocated 71720 bytes for local input buffer
MERGE Data Link (25675): File mapped object /SHM_11014 accessed
MERGE Data Link (25675): Memory mapping 100669480 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef3d11e000.
MERGE Data Link (25676): Message logger not contacted.
MERGE Data Link (25676): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25676): Configuration: index = 14, SHM key=11000, TCP port base = 11015, module = 13, hardware = 0
MERGE Data Link (25676): File mapped object /SHM_11000 accessed
MERGE Data Link (25676): Memory mapping 536628 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6aea271000.
MERGE Data Link (25676): Allocated 71720 bytes for local input buffer
MERGE Data Link (25676): File mapped object /SHM_11015 accessed
MERGE Data Link (25676): Memory mapping 100669480 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6ae426f000.
MERGE Data Link (25677): Message logger not contacted.
MERGE Data Link (25677): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25677): Configuration: index = 15, SHM key=11000, TCP port base = 11016, module = 14, hardware = 0
MERGE Data Link (25677): File mapped object /SHM_11000 accessed
MERGE Data Link (25677): Memory mapping 536628 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4281e9000.
MERGE Data Link (25677): Allocated 71720 bytes for local input buffer
MERGE Data Link (25677): File mapped object /SHM_11016 accessed
MERGE Data Link (25677): Memory mapping 100669480 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4221e7000.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
MERGE Statistics thread starting
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
... 201 more lines ...
Attachment 3: 210602_TapeServerStartup
Tidy up
master(4596): Operation not permitted
Starting Tape Server

[1] 25639
MIDAS Tape Server: Message logger not contacted.
MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021  12:22:14
MIDAS Tape Server: Unable to change scheduling priority - Permission denied
MIDAS Tape Server: Using default startup
MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205.
MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19721000.
MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration
MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats
MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305                 
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002                
MIDAS Tape Server: Message reporting level = 0x180fff8
MIDAS Tape Server: Message logging level = 0xfff8
MIDAS Tape Server: Tape Server Options = 0x0
MIDAS Tape Server: File device path base = /TapeData
MIDAS Tape Server: Data buffer size = 65536
MIDAS Tape Server: Tape block size = 65536
MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19320000.
MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19881000.
MIDAS Tape Server: Capabilities restored.
MIDAS Tape Server: Master global area initialised.
MIDAS Tape Server: Stats task has pid 25641
MIDAS Tape Server: Driver process for /dev/file/0 has pid 25642
MIDAS Tape Server: Driver process for /dev/file/1 has pid 25643
MIDAS Tape Server: Driver process for /dev/null/0 has pid 25644
MIDAS Tape Server: Link task 0 has pid 25645
MIDAS Tape Server: Link task 1 has pid 25646
MIDAS Tape Server: Starting the RPC interface
MIDAS Tape Statistics: Message logger not contacted.
MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021  12:22:20
MIDAS Tape Statistics: Started with args 10205
MIDAS Tape Statistics: Configuration: SHM key=10205
MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Statistics: Shared memory ID is 3
MIDAS Tape Statistics: Shared memory segment located at address 7f9c551f7000.
MIDAS Data Link (25645): Message logger not contacted.
MIDAS Data Link (25645): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (25645): Started with args 10205 10305                 
MIDAS Data Link (25645): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Data Link (25646): Message logger not contacted.
MIDAS Tape Driver (25644): Started with args 2 10205
MIDAS Data Link (25645): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25645): Shared memory ID is 3
MIDAS Data Link (25645): Shared memory segment located at address 7f6ba30ef000.
MIDAS Data Link (25646): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Tape Driver (25644): Configuration: driver=2, key=10205.
MIDAS Data Link (25646): Started with args 10205 21001 21002                
MIDAS Data Link (25645): Starting the network interface
MIDAS Tape Driver (25644): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2
MIDAS Tape Driver (25644): Shared memory ID is 3
MIDAS Tape Driver (25644): Shared memory segment located at address 7f09d718d000.
MIDAS Tape Driver (25644): Using device /dev/null/0 of type sink.
MIDAS Data Link (25646): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Shared memory ID is 3
MIDAS Data Link (25646): Shared memory segment located at address 7fa75ae3c000.
MIDAS Data Link (25646): Starting the network interface
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Data Link (25645): TCP socket receive buffer was 87380 - now 249856
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Tape Driver (25642): Started with args 0 10205
MIDAS Tape Driver (25642): Configuration: driver=0, key=10205.
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): Shared memory ID is 3
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25645): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Started with args 1 10205
MIDAS Tape Driver (25642): Shared memory segment located at address 7f48b1ea4000.
MIDAS Tape Driver (25642): Using device /dev/file/0 of type file.
MIDAS Data Link (25645): MIDAS Data Link thread 0 using TCP port 10305.
MIDAS Data Link (25645): Entering server loop
MIDAS Tape Driver (25643): Configuration: driver=1, key=10205.
MIDAS Data Link (25645): thread 0 listening on port 10305
MIDAS Tape Driver (25643): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Shared memory ID is 3
MIDAS Tape Driver (25643): Shared memory segment located at address 7f6044052000.
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Using device /dev/file/1 of type file.
MIDAS Data Link (25646): MIDAS Data Link thread 0 using TCP port 21001.
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (25646): MIDAS Data Link thread 1 using TCP port 21002.
MIDAS Data Link (25646): Entering server loop
MIDAS Data Link (25646): thread 0 listening on port 21001
MIDAS Data Link (25646): thread 1 listening on port 21002
MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205.
MIDAS Tape Server: Entering server loop
MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205.
MIDAS Data Link (25645): thread 0 accepted connection from 127.0.0.1, port 37330
MIDAS Data Link (25645): buffer size changed to 65536
MIDAS Tape Server: device /dev/file/0 allocated.
MIDAS Tape Server: Mounting volume S496 on device /dev/file/0.
MIDAS Tape Server: Opening file R31 on device /dev/file/0.
2222 64 0 31056336
3333 31056336 2048000 1 15 

Attachment 4: TS_configuration
#device configuration information
#    any line starting with a # is a comment and is ignored
stats /MIDAS/TapeServer/Linux64/stats
#    devices available   -   list ends with a null line
#    format       class_name     device_name    driver_task_path_name
#       class_name MUST be one of dlt, exabyte, scsitape, sink
file          /dev/file/0    /MIDAS/TapeServer/Linux64/driver
file          /dev/file/1    /MIDAS/TapeServer/Linux64/driver
sink          /dev/null/0     /MIDAS/TapeServer/Linux64/driver

#data link configuration information  - list end with a null line
#    format   link_task_path_name  
/MIDAS/TapeServer/Linux64/linkTCP 10305
/MIDAS/TapeServer/Linux64/linkTCP 21001 21002

#program options - list end with a null line
#tapeserver_options 0x2
tapeserver_options 0x0
msg_reporting_level 0x0180fff8
#   use 0x0080fff8 to enable msg logging
msg_logging_level 0xfff8
#    default for following is 16Kbytes - both MUST be the same at present
data_buffer_size 64
tape_block_size 64
disc_file_size 2000
#
file_path_base  /TapeData

#end of information


Attachment 5: CONTENTS
Index string MERGE.LinksAvailable&&RunNumber&&MERGE.LinksInUse&&MERGE.RunOptions
Stat.shift string 0
0x0006dead string 0x0000
MERGE.LinksAvailable string 16
Stat.offset string 0
RunNumber string 0
Rate.channels string 1024
Stat.channels string 1024
MERGE.RunOptions string 3
MERGE.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Merge.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%
Attachment 6: 210602_merger.png
210602_merger.png
  352   Sun May 30 08:57:53 2021 TDSunday 30 May
09.55 DAQ continues - file S496/R18_588
      alpha background

      increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e98 : 	 5  
aida06 fault 	 0x545a : 	 0x545f : 	 5  
aida07 fault 	 0xc04d : 	 0xc052 : 	 5  
aida08 fault 	 0xa0f7 : 	 0xa0fc : 	 5  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      6      9      5      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      5      5      2      2      4      3      3      2      4      6   : 37956
aida03 :     22     12      7      5      3      3      2      3      3      3      6   : 36872
aida04 :     40     13      5      5      3      4      2      3      3      3      6   : 37048
aida05 :     15      5      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     11      6      2      1      2      3      3      4      2      4      6   : 38300
aida07 :     17      5      4      3      1      3      3      2      2      4      6   : 37324
aida08 :      1      8      1      2      2      4      4      3      2      4      6   : 38164
aida09 :     10      8      6      1      2      4      3      3      1      4      6   : 36968
aida10 :     14      7      5      2      1      3      3      3      1      4      6   : 36800
aida11 :      7      9      3      1      2      3      3      3      1      4      6   : 36788
aida12 :      3      5      2      3      1      4      2      3      2      4      6   : 37620
aida13 :     12      4      4      2      1      3      2      2      2      4      6   : 37008
aida14 :     11     10      5      1      1      3      2      2      2      4      6   : 37036
aida15 :     11      5      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     10      4      8      1      2      3      2      2      2      4      6   : 37096

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.02% aida03

18.40 DAQ continues - file S496/R18_589
      alpha background

      increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-30_Statistics_aidas-gsi.png
Screenshot_2021-05-30_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 960.png
960.png
Attachment 8: 961.png
961.png
Attachment 9: R18_589
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      31985
          ADC data format:   58630511 (   25364.6 Hz)
        Other data format:  203166719 (   87893.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         35 (       0.0 Hz)
                               RESUME:         35 (       0.0 Hz)
                              SYNC100:      40596 (      17.6 Hz)
                              WR48-63:      40596 (      17.6 Hz)
                           FEE64 disc:    1341777 (     580.5 Hz)
                             MBS info:  201743680 (   87277.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2311.511 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             1879.226
  2                0.365                0.000
  3                0.313                0.000
  4                0.000             1697.189
  5                0.000              557.330
  6                0.000             1686.560
  7                0.000              647.171
  8                0.000                0.000
  9                0.000                0.000
 10                0.382                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        157          0          0          0          0          0          0          0          0          0          0          0
  1    3314609       1296          0          0          0          0        648        648          0          0          0          0
  2        413   87031526          0          0         19         19      14856      14856          0   87001776          0          0
  3   42258127   88383449          0          0         15         15      18954      18954    1341775   87003736          0          0
  4        153    6936262          0          0          0          0        860        860          0    6934542          0          0
  5    6312304    6939162          0          0          0          0       2310       2310          0    6934542          0          0
  6        544    6936170          0          0          0          0        814        814          0    6934542          0          0
  7    6741331    6938850          0          0          0          0       2154       2154          0    6934542          0          0
  8        448          0          0          0          0          0          0          0          0          0          0          0
  9         79          0          0          0          0          0          0          0          0          0          0          0
 10       1189          2          0          0          1          1          0          0          0          0          0          0
 11        111          2          0          0          0          0          0          0          2          0          0          0
 12        767          0          0          0          0          0          0          0          0          0          0          0
 13         99          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        180          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.504s ( 1015.271 blocks/s,  63.454 Mb/s)
  351   Sat May 29 13:13:45 2021 TDSaturday 29 May
14.10 DAQ continues - file S496/R18_539
      alpha background

      increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e97 : 	 4  
aida06 fault 	 0x545a : 	 0x545e : 	 4  
aida07 fault 	 0xc04d : 	 0xc051 : 	 4  
aida08 fault 	 0xa0f7 : 	 0xa0fb : 	 4  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      7      2      2      2      2      2      2      4      6   : 37036
aida02 :     14      3      5      2      2      4      3      3      2      4      6   : 37984
aida03 :     15     11      8      6      3      3      2      3      3      3      6   : 36884
aida04 :     43     13      5      5      3      4      2      3      3      3      6   : 37060
aida05 :     19      3      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     17      4      1      3      2      3      3      4      2      4      6   : 38356
aida07 :     16      6      2      1      0      2      2      3      2      4      6   : 37296
aida08 :     21      8      1      1      3      4      4      3      2      4      6   : 38276
aida09 :     13     10      2      1      2      3      2      2      2      4      6   : 37060
aida10 :     13      9      4      1      1      3      3      3      1      4      6   : 36764
aida11 :     13      9      2      1      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :      7      6      6      2      1      3      2      2      2      4      6   : 37036
aida14 :      9      7      7      1      1      3      2      2      2      4      6   : 37036
aida15 :     14      5      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.12% aida04

18.40 DAQ continues - file S496/R18_551
      alpha background

      increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-29_Statistics_aidas-gsi.png
Screenshot_2021-05-29_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 950.png
950.png
Attachment 8: 951.png
951.png
Attachment 9: R18_541
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   46792076 (   18702.9 Hz)
        Other data format:  215127929 (   85987.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         89 (       0.0 Hz)
                               RESUME:         89 (       0.0 Hz)
                              SYNC100:      41318 (      16.5 Hz)
                              WR48-63:      41318 (      16.5 Hz)
                           FEE64 disc:    1013016 (     404.9 Hz)
                             MBS info:  214032099 (   85549.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2501.862 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             1964.282
  2                0.467                0.000
  3                3.050                0.000
  4                0.000             1832.921
  5                0.000              844.521
  6                0.000             1829.325
  7                0.000              869.373
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        499          0          0          0          0          0          0          0          0          0          0          0
  1      74891         24          0          0          0          0         12         12          0          0          0          0
  2       1032   92085039          0          0         27         27      16251      16251          0   92052483          0          0
  3   37271457   93008741          0          0         62         62      19135      19135    1013015   91957332          0          0
  4        552    7507421          0          0          0          0        925        925          0    7505571          0          0
  5    4051453    7509613          0          0          0          0       2021       2021          0    7505571          0          0
  6       1401    7507471          0          0          0          0        950        950          0    7505571          0          0
  7    5387708    7509619          0          0          0          0       2024       2024          0    7505571          0          0
  8        460          0          0          0          0          0          0          0          0          0          0          0
  9        183          0          0          0          0          0          0          0          0          0          0          0
 10        537          0          0          0          0          0          0          0          0          0          0          0
 11        388          0          0          0          0          0          0          0          0          0          0          0
 12        674          0          0          0          0          0          0          0          0          0          0          0
 13        244          1          0          0          0          0          0          0          1          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        597          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.855s ( 1004.537 blocks/s,  62.784 Mb/s)
  350   Fri May 28 10:29:07 2021 TDFriday 28 May
11.30 DAQ continues - file S496/R18_472
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      7      1      3      2      2      2      2      4      6   : 36980
aida02 :      3      1      5      1      2      4      3      3      2      4      6   : 37892
aida03 :      8      8      6      6      3      3      2      3      3      3      6   : 36800
aida04 :     37     12      5      6      3      4      2      3      3      3      6   : 37060
aida05 :     11      5      6      4      2      2      2      2      2      4      6   : 37044
aida06 :      4      3      3      3      2      2      2      3      3      4      6   : 38456
aida07 :      1     12      3      2      0      3      3      2      2      4      6   : 37204
aida08 :     13      8      1      2      3      4      3      3      2      4      6   : 38020
aida09 :      7      7      5      1      1      3      2      2      2      4      6   : 36996
aida10 :     12      7      9      2      1      3      3      3      1      4      6   : 36856
aida11 :      2      9      3      1      2      3      3      3      1      4      6   : 36768
aida12 :     11      4      3      3      2      4      2      3      2      4      6   : 37724
aida13 :     10      8      6      2      2      3      2      2      2      4      6   : 37128
aida14 :     16      7      7      1      0      4      2      2      2      4      6   : 37128
aida15 :     12      6      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_471 - attachment 9
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-28_Statistics_aidas-gsi.png
Screenshot_2021-05-28_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 940.png
940.png
Attachment 8: 941.png
941.png
Attachment 9: R18_471
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57407666 (   20753.5 Hz)
        Other data format:  204512336 (   73933.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        116 (       0.0 Hz)
                               RESUME:        116 (       0.0 Hz)
                              SYNC100:      42303 (      15.3 Hz)
                              WR48-63:      42303 (      15.3 Hz)
                           FEE64 disc:    1160567 (     419.6 Hz)
                             MBS info:  203266931 (   73483.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        109 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2766.172 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2700.364
  2                0.332                0.000
  3                4.856                0.000
  4                0.000                0.000
  5                0.000             1685.309
  6                0.000                0.000
  7                0.000             1567.327
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        181         23          0          0          0          0          0          0         23          0          0         13
  1     838470        386          0          0          0          0        166        166         54          0          0          0
  2        162  101753181          0          0         19         19      17931      17931         15  101717266          0          5
  3   42392398  102752755          0          0         97         97      21331      21331    1160234  101549665          0         12
  4        211         21          0          0          0          0          0          0         21          0          0         10
  5    6478076       2740          0          0          0          0       1337       1337         66          0          0          0
  6        260         17          0          0          0          0          0          0         17          0          0          4
  7    7695649       3163          0          0          0          0       1537       1537         89          0          0         28
  8        132         14          0          0          0          0          1          1         12          0          0         10
  9        368          5          0          0          0          0          0          0          5          0          0          4
 10        173         18          0          0          0          0          0          0         18          0          0          5
 11        303          5          0          0          0          0          0          0          5          0          0          5
 12        186          1          0          0          0          0          0          0          1          0          0          7
 13        585          2          0          0          0          0          0          0          2          0          0          2
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        512          5          0          0          0          0          0          0          5          0          0          4
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.742s ( 1008.122 blocks/s,  63.008 Mb/s)
  349   Thu May 27 12:25:53 2021 TDThursday 27 May
13.29 DAQ continues - file S496/R18_443
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      7      7      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      1      5      1      3      4      3      3      2      4      6   : 37956
aida03 :      2      6      5      6      2      4      2      3      3      3      6   : 36808
aida04 :     24     12      5      5      4      3      2      3      3      3      6   : 36912
aida05 :      8      1      9      3      2      2      2      2      2      4      6   : 37016
aida06 :      3      6      2      2      2      2      2      3      3      4      6   : 38428
aida07 :      6     17      1      2      0      3      3      2      2      4      6   : 37232
aida08 :      4      2      1      2      3      4      3      3      2      4      6   : 37936
aida09 :     15      7      5      2      3      4      3      3      1      4      6   : 37060
aida10 :      7      6      9      2      0      3      3      3      1      4      6   : 36764
aida11 :     19     10      2      0      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :     17      9      2      0      0      4      2      2      2      4      6   : 37036
aida14 :     15      4      7      1      1      3      2      2      2      4      6   : 37036
aida15 :      9      6      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     11     11      8      2      0      3      2      2      2      4      6   : 37060

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_442 - attachment 9
       max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-27_Statistics_aidas-gsi.png
Screenshot_2021-05-27_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 930.png
930.png
Attachment 8: 931.png
931.png
Attachment 9: R18_442
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   68693190 (   21525.3 Hz)
        Other data format:  193226813 (   60548.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.0 Hz)
                               RESUME:        114 (       0.0 Hz)
                              SYNC100:      43889 (      13.8 Hz)
                              WR48-63:      43889 (      13.8 Hz)
                           FEE64 disc:    1335474 (     418.5 Hz)
                             MBS info:  191803333 (   60102.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          3 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3191.275 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2989.890
  2                0.511              643.077
  3                8.760                0.000
  4                0.000                0.000
  5                0.000             1578.752
  6                0.000                0.000
  7                0.000             1340.150
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        155          0          0          0          0          0          0          0          0          0          0          0
  1     527962        216          0          0          0          0        108        108          0          0          0          0
  2        533   96091128          0          0         28         28      16832      16832          0   96057408          0          0
  3   49735236   97127126          0          0         86         86      22778      22778    1335473   95745925          0          0
  4        118          2          0          0          0          0          1          1          0          0          0          0
  5    8268444       3874          0          0          0          0       1937       1937          0          0          0          0
  6        632          0          0          0          0          0          0          0          0          0          0          0
  7   10156762       4466          0          0          0          0       2233       2233          0          0          0          0
  8        500          0          0          0          0          0          0          0          0          0          0          0
  9        102          0          0          0          0          0          0          0          0          0          0          0
 10       1498          0          0          0          0          0          0          0          0          0          0          0
 11        105          0          0          0          0          0          0          0          0          0          0          0
 12        886          0          0          0          0          0          0          0          0          0          0          0
 13         95          1          0          0          0          0          0          0          1          0          0          2
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        162          0          0          0          0          0          0          0          0          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.027s (  999.146 blocks/s,  62.447 Mb/s)
  348   Wed May 26 13:21:59 2021 TDWednesday 26 May
14.22 DAQ continues - file S496/R18_369
      alpha background

      All system wide checks OK *except*

	
		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     12      4      6      1      2      3      2      2      2      4      6   : 37072
aida02 :     12      2      4      1      3      4      3      3      2      4      6   : 37984
aida03 :     12      6      7      6      2      4      3      2      3      3      6   : 36624
aida04 :     36     11      4      6      4      3      2      3      3      3      6   : 36968
aida05 :      2      4      6      3      2      2      2      2      2      4      6   : 36968
aida06 :      8      5      3      2      2      2      2      3      3      4      6   : 38456
aida07 :     10      4      6      4      3      3      3      3      2      4      6   : 37992
aida08 :     21      8      3      2      3      3      3      3      2      4      6   : 37956
aida09 :     17     10      3      2      3      4      3      3      1      4      6   : 37060
aida10 :      3      6     10      2      1      3      3      3      1      4      6   : 36828
aida11 :     12      6      4      2      2      3      3      3      1      4      6   : 36832
aida12 :      2      9      5      1      3      3      2      3      2      4      6   : 37632
aida13 :      4      7      2      1      3      3      3      3      2      4      6   : 37832
aida14 :     15      6      6      1      0      4      2      2      2      4      6   : 37100
aida15 :      5      4      3      2      1      3      3      4      2      4      6   : 38244
aida16 :     17      6      6      2      1      3      2      2      2      4      6   : 37076

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_340 - attachment 9
       max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-26_Statistics_aidas-gsi.png
Screenshot_2021-05-26_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 920.png
920.png
Attachment 8: 921.png
921.png
Attachment 9: R18_368
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   60865731 (   22299.7 Hz)
        Other data format:  201054280 (   73661.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        152 (       0.1 Hz)
                               RESUME:        152 (       0.1 Hz)
                              SYNC100:      42166 (      15.4 Hz)
                              WR48-63:      42166 (      15.4 Hz)
                           FEE64 disc:    1309767 (     479.9 Hz)
                             MBS info:  199659877 (   73150.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        107 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2729.447 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2401.079
  2                0.453                0.000
  3                8.570                0.000
  4                0.000                0.000
  5                0.000             1507.937
  6                0.000                0.000
  7                0.000             1499.170
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0       1115         29          0          0          0          0          1          1         27          0          0         10
  1    2766811      11645          0          0          0          0        574        574      10497          0          0         18
  2       2194  100014163          0          0         27         27      17354      17354          0   99979401          0          1
  3   44009553  100997602          0          0        125        125      21286      21286    1274310   99680470          0          5
  4        902          6          0          0          0          0          0          0          0          6          0          0
  5    6872755      14651          0          0          0          0       1451       1451      11749          0          0         10
  6       1948         13          0          0          0          0          1          1         11          0          0          9
  7    7202790      16106          0          0          0          0       1497       1497      13112          0          0          9
  8        531          0          0          0          0          0          0          0          0          0          0          0
  9        394          0          0          0          0          0          0          0          0          0          0          3
 10       1118          1          0          0          0          0          0          0          1          0          0          0
 11        900         26          0          0          0          0          0          0         26          0          0         12
 12       1756         13          0          0          0          0          0          0         13          0          0          8
 13       1346         11          0          0          0          0          1          1          9          0          0          7
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15       1618         14          0          0          0          0          1          1         12          0          0         15
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.695s ( 1009.613 blocks/s,  63.101 Mb/s)
  347   Tue May 25 11:16:54 2021 TDTuesday 25 May
12.16 DAQ continues file S496/R18_324
      alpha background

      Tape server c. 2.5Mb/s
      ASIC check
      Tape server c. 300kb/s - presumably hot HEC channels

12.20 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      6      2      1      3      2      2      2      4      6   : 37016
aida02 :      3      6      5      3      3      4      3      2      1      3      7   : 38572
aida03 :     21     10      8      6      3      3      2      3      2      4      6   : 37924
aida04 :     29     15      4      5      3      4      2      3      3      3      6   : 37004
aida05 :     17      5      2      2      2      2      2      2      2      4      6   : 36940
aida06 :      1      6      0      1      2      2      2      3      3      4      6   : 38356
aida07 :     11      6      7      4      3      3      3      3      2      4      6   : 38028
aida08 :     13      8      3      2      2      3      3      2      1      3      7   : 38372
aida09 :      7      9      2      1      3      3      3      3      2      4      6   : 37860
aida10 :      9      8      4      2      0      3      3      3      1      4      6   : 36708
aida11 :     16      6      3      2      2      3      3      3      1      4      6   : 36832
aida12 :     21     10      7      4      2      4      2      2      1      3      7   : 38420
aida13 :      8      3      3      1      3      3      3      3      2      4      6   : 37832
aida14 :     14      3      6      1      0      4      2      2      2      4      6   : 37072
aida15 :      2      4      2      2      1      3      3      4      2      4      6   : 38216
aida16 :     16      5      5      3      0      3      2      2      2      4      6   : 37016


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Rate spectra - attachment 2
      ADC data item stats - attachment 3
      FEE64 temps OK - attachment 4
      DSSSD bias & leakage current OK - attachment 5
      Merger/Tape Server/Merger stats - attachment 6
       no merger errors reported since previous restart

17.30 analysis of file S496/R18_340
       max deadtime 0.36% aida04
Attachment 1: Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-05-25_Statistics_aidas-gsi.png
Screenshot_2021-05-25_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: 910.png
910.png
Attachment 6: 911.png
911.png
Attachment 7: R18_340
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   63855540 (   23748.1 Hz)
        Other data format:  198064462 (   73660.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        109 (       0.0 Hz)
                               RESUME:        109 (       0.0 Hz)
                              SYNC100:      42015 (      15.6 Hz)
                              WR48-63:      42015 (      15.6 Hz)
                           FEE64 disc:    1291036 (     480.1 Hz)
                             MBS info:  196689178 (   73149.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:         35 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2688.871 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2304.866
  2                0.344                0.000
  3                9.625                0.000
  4                0.000                0.000
  5                0.000             1450.665
  6                0.000                0.000
  7                0.000             1243.447
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        303          7          0          0          0          0          1          1          5          0          0          5
  1    3031185       1240          0          0          0          0        620        620          0          0          0          0
  2        536   98548687          0          0         20         20      16872      16872          2   98514901          0          3
  3   44821969   99507955          0          0         89         89      21248      21248    1291004   98174277          0          0
  4        514          3          0          0          0          0          1          1          1          0          0          1
  5    7529304       3056          0          0          0          0       1528       1528          0          0          0          0
  6       1314          2          0          0          0          0          0          0          2          0          0          2
  7    8467764       3488          0          0          0          0       1744       1744          0          0          0          0
  8        226          4          0          0          0          0          0          0          4          0          0          4
  9        144          4          0          0          0          0          0          0          4          0          0          4
 10        524          1          0          0          0          0          0          0          1          0          0          0
 11        432          2          0          0          0          0          0          0          2          0          0          2
 12        719          0          0          0          0          0          0          0          0          0          0          0
 13        154          7          0          0          0          0          1          1          5          0          0          7
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        452          6          0          0          0          0          0          0          6          0          0          7
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   39.492s (  810.287 blocks/s,  50.643 Mb/s)
  346   Mon May 24 09:57:07 2021 TDMonday 24 May
07.30 Hot HEC channels multiple FEE64s
      ASIC check fixes

10.56 DAQ continues OK file S496/R18_75

      System wide checks OK * except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      2      7      2      1      4      3      2      1      4      7   : 40452
aida02 :     13      3      3      1      3      4      2      2      2      4      7   : 41308
aida03 :     17      6      2      2      1      4      2      3      2      3      7   : 39700
aida04 :      8      6      1      0      1      3      1      4      2      3      7   : 39712
aida05 :     10      3      5      3      4      3      3      2      1      4      7   : 40560
aida06 :      4      4      2      0      2      3      2      3      3      4      7   : 42576
aida07 :      5      3      2      3      1      3      1      3      2      4      7   : 41324
aida08 :     11      5      1      1      2      3      3      2      2      4      7   : 41348
aida09 :      7      4      3      0      2      3      1      3      2      4      7   : 41324
aida10 :     11      4      2      2      2      3      2      4      1      4      7   : 41132
aida11 :     18      5      0      2      2      4      2      2      1      4      7   : 40240
aida12 :      2      3      0      1      2      3      2      2      2      4      7   : 41024
aida13 :      4      5      1      2      2      3      2      4      1      4      7   : 41096
aida14 :      9      3      1      2      1      4      2      2      2      4      7   : 41164
aida15 :      2      3      1      3      2      2      3      3      2      4      7   : 41744
aida16 :      5      5      2      0      3      3      2      4      1      4      7   : 41116

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
      1.8.H spectra - attachments 6-8
      1.8.L spectra - attachments 9 & 10
      Stat spectra - attachments 11 & 12
      1.8.W spectra - 20us FSR - attachments 13 & 14
      ADC data items - attachment 15
      FEE64 temperatures OK - attachment 16
      DSSSD bias & leakage currents OK - attachment 17
      Merger/Tape Server/Merger statistics - attachment 18
       no merger errors reported since previous restart

      analysis of file R18_74
       max dead time 0.05%

10.17 ASIC check
      All histograms, statistics, merger statistics zero'd
Attachment 1: Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-24_Statistics_aidas-gsi(4).png
Screenshot_2021-05-24_Statistics_aidas-gsi(4).png
Attachment 3: Screenshot_2021-05-24_Statistics_aidas-gsi(3).png
Screenshot_2021-05-24_Statistics_aidas-gsi(3).png
Attachment 4: Screenshot_2021-05-24_Statistics_aidas-gsi(2).png
Screenshot_2021-05-24_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-24_Statistics_aidas-gsi(1).png
Screenshot_2021-05-24_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png
Attachment 7: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png
Attachment 8: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png
Attachment 9: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png
Attachment 10: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png
Attachment 12: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png
Attachment 13: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png
Attachment 14: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-05-24_Statistics_aidas-gsi.png
Screenshot_2021-05-24_Statistics_aidas-gsi.png
Attachment 16: Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: 900.png
900.png
Attachment 18: 901.png
901.png
Attachment 19: R18_74
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   42982076 (   14466.2 Hz)
        Other data format:  218937925 (   73686.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         59 (       0.0 Hz)
                               RESUME:         59 (       0.0 Hz)
                              SYNC100:      43069 (      14.5 Hz)
                              WR48-63:      43069 (      14.5 Hz)
                           FEE64 disc:    1163669 (     391.6 Hz)
                             MBS info:  217688000 (   73265.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2971.209 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.425                0.000
  3                1.488                0.000
  4                0.000                0.000
  5                0.000             2622.333
  6                0.000                0.000
  7                0.000             2340.476
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         64          0          0          0          0          0          0          0          0          0          0          0
  1       2227          3          0          0          0          0          1          1          1          0          0          0
  2         75  108903724          0          0         25         25      19964      19964          0  108863746          0          0
  3   35357926  110031040          0          0         34         34      21525      21525    1163668  108824254          0          0
  4         89          0          0          0          0          0          0          0          0          0          0          0
  5    3145189       1316          0          0          0          0        658        658          0          0          0          0
  6         64          0          0          0          0          0          0          0          0          0          0          0
  7    4475895       1842          0          0          0          0        921        921          0          0          0          0
  8         56          0          0          0          0          0          0          0          0          0          0          0
  9         67          0          0          0          0          0          0          0          0          0          0          0
 10        115          0          0          0          0          0          0          0          0          0          0          0
 11         65          0          0          0          0          0          0          0          0          0          0          0
 12         65          0          0          0          0          0          0          0          0          0          0          0
 13         77          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        102          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   40.766s (  784.975 blocks/s,  49.061 Mb/s)
  345   Sat May 22 23:40:32 2021 TDSunday 23 May
00.40 Unable to connect to FEE64s 1-7
      DAQ power cycled OK

      ASIC settings 2021Apr29-13-16-00
      slow comparator 0x64 *all* FEE64s

      BNC PB-5 OFF

      file S496/R16

      alpha background

07.17 Some hot HEC channels
      ASIC check fixes issue
      System wide checks OK *except*

		 Base 		Current 	Difference
aida06 fault 	 0x5457 : 	 0x5457 : 	 0  
aida06 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1cd : 	 461  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      3      1      4      3      2      1      4      7   : 40552
aida02 :      3      2      2      1      2      3      3      2      2      4      7   : 41308
aida03 :     14      5      0      1      1      3      2      4      2      3      7   : 40000
aida04 :     14      4      5      1      3      3      1      4      2      3      7   : 39944
aida05 :     12      4      1      3      2      3      3      2      2      4      7   : 41408
aida06 :      8      3      2      1      2      4      2      3      3      4      7   : 42744
aida07 :     10      4      3      2      1      2      2      3      2      4      7   : 41464
aida08 :      7      5      2      0      3      3      3      2      2      4      7   : 41380
aida09 :     10      5      1      1      1      2      2      3      2      4      7   : 41408
aida10 :     11      6      3      2      1      3      2      2      2      4      7   : 41100
aida11 :     12      8      3      2      2      2      3      2      1      4      7   : 40288
aida12 :      9      9      3      1      2      3      3      2      1      4      7   : 40380
aida13 :     10      4      0      1      1      2      1      3      2      4      7   : 41128
aida14 :      9      4      2      1      1      2      1      3      2      4      7   : 41156
aida15 :      5      4      0      3      3      3      3      3      2      4      7   : 41940
aida16 :      9      6      3      2      0      2      1      3      2      4      7   : 41156

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 00:29:48 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      FEE64 temperatures OK - attachment 1
      ADC data item stats - attachment 2
      Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
      DSSSD bias & leakage currents OK - attachment 4

11.18 Analysis of file S496/R16_33 - attachment 5

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  685M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.0T  867G  88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

867/2 x 3279s = 16.5 days to filesystem full - OK

12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
      aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK

      DAQ continues file S496/R18
Attachment 1: Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-23_Statistics_aidas-gsi.png
Screenshot_2021-05-23_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: 810.png
810.png
Attachment 5: R16_33
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   47385414 (   14450.2 Hz)
        Other data format:  214534594 (   65422.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         61 (       0.0 Hz)
                               RESUME:         61 (       0.0 Hz)
                              SYNC100:      44215 (      13.5 Hz)
                              WR48-63:      44215 (      13.5 Hz)
                           FEE64 disc:    1311957 (     400.1 Hz)
                             MBS info:  213134085 (   64995.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3279.230 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.386                0.000
  3                1.000                0.000
  4                0.000                0.000
  5                0.000             2849.824
  6                0.000                0.000
  7                0.000             2608.689
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         71          0          0          0          0          0          0          0          0          0          0          0
  1       1455          0          0          0          0          0          0          0          0          0          0          0
  2         87  106617422          0          0         35         35      20186      20186          0  106576980          0          0
  3   39184250  107913590          0          0         26         26      22238      22238    1311957  106557105          0          0
  4         63          0          0          0          0          0          0          0          0          0          0          0
  5    3345685       1512          0          0          0          0        756        756          0          0          0          0
  6         60          0          0          0          0          0          0          0          0          0          0          0
  7    4853274       2070          0          0          0          0       1035       1035          0          0          0          0
  8         69          0          0          0          0          0          0          0          0          0          0          0
  9         45          0          0          0          0          0          0          0          0          0          0          0
 10         70          0          0          0          0          0          0          0          0          0          0          0
 11         68          0          0          0          0          0          0          0          0          0          0          0
 12         89          0          0          0          0          0          0          0          0          0          0          0
 13         57          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15         71          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.293s (  722.462 blocks/s,  45.154 Mb/s)
  344   Sat May 22 13:40:56 2021 TDSaturday 22 May
14.37 DAQ continues file S496/R15_14

      Grafana - DSSSD bias & leakage current - attachment 1
      1.8.L spectra - attachments 2 & 3
      Stat spectra - attachments 4 & 5
      FEE64 temperatures OK - attachment 6
      stats - ADC data items, pause items OK - attachments 7 & 8
      DSSSD bias & leakage current OK - attachment 9
      Merger/Tape Server/Merger stats - attachment 10
      analysis of file S496/R14_1920 - attachment 11
       all FEE64 deadtime 0% as expected

17.30 aida01, 3, 5, 9 & 11 have hot HEC channels
       ASIC check fixes
      All histogram, statistics & merger statistics zero'd

      DAQ continues
      Note gamma-ray source tests have started with FATIMA and DEGAS

20.20 Analysis file R15_19 - attachment 12

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  665M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.3T  530G  93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

        estimate 90/2 x 3168s = 40h to filesystem full - OK

20.22 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xf82c : 	 0xf82f : 	 3  
aida02 fault 	 0xdda6 : 	 0xdda9 : 	 3  
aida03 fault 	 0xb13d : 	 0xb140 : 	 3  
aida04 fault 	 0xf4cd : 	 0xf4d0 : 	 3  
aida05 fault 	 0x500 : 	 0x575 : 	 117  
aida06 fault 	 0xc6e7 : 	 0xc6e9 : 	 2  
aida07 fault 	 0x4920 : 	 0x4922 : 	 2  
aida08 fault 	 0xbd9a : 	 0xbd9c : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      8      4      6      6      1      4      3      3      4      3      6   : 38048
aida02 :     11      8     10      3      4      4      3      4      3      3      6   : 37740
aida03 :      2      8      3      3      2      2      1      4      2      4      6   : 37720
aida04 :     26     15     17      4      4      2      1      4      2      4      6   : 38256
aida05 :     15      6      4      1      1      2      1      3      3      4      6   : 38156
aida06 :      7      7     16      6      3      4      2      2      3      4      6   : 38612
aida07 :     11      5     12      3      3      3      2      3      1      3      7   : 38836
aida08 :     13      6      7      2      1      2      2      2      3      4      6   : 37972
aida09 :     23      3      8      5      0      5      3      3      4      3      6   : 38164
aida10 :     12      5      7      1      2      2      4      3      2      4      6   : 37992
aida11 :     19      2      7      1      2      2      1      2      3      4      6   : 37740
aida12 :     21      5      3      3      3      3      1      2      3      4      6   : 37964
aida13 :     18      7      7      1      2      4      2      3      2      4      6   : 37776
aida14 :      8      3     13      7      2      4      2      3      2      4      6   : 37992
aida15 :     11      5     14      1      3      4      2      3      2      4      6   : 37908
aida16 :     17      6      6      2      2      3      1      2      3      4      6   : 37908

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.30 Lost activity monitors - attachments 13-15

20.54 1.8.W spectra 20us FSR attachments 16 & 23
      1.8.W spectra 200us FSR attachments 17 & 22
      1.8.W spectra 2ms FSR attachments 18 & 21
      1.8.W spectra 20ms FSR attachments 19 & 20
Attachment 1: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png
Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png
Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png
Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png
Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png
Attachment 6: Screenshot_2021-05-22_Statistics_aidas-gsi(9).png
Screenshot_2021-05-22_Statistics_aidas-gsi(9).png
Attachment 7: Screenshot_2021-05-22_Statistics_aidas-gsi(8).png
Screenshot_2021-05-22_Statistics_aidas-gsi(8).png
Attachment 8: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 9: 805.png
805.png
Attachment 10: 806.png
806.png
Attachment 11: R15_13
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  102550190 (   10129.0 Hz)
        Other data format:  159369822 (   15741.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      69714 (       6.9 Hz)
                              WR48-63:      69714 (       6.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:  159230394 (   15727.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:    10124.450 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000               67.951
  3                0.000                0.000
  4                0.000                0.000
  5                0.000             9971.491
  6                0.000                0.000
  7                0.000            10069.837
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        213          0          0          0          0          0          0          0          0          0          0          0
  1       1159          2          0          0          0          0          1          1          0          0          0          0
  2        279   79660137          0          0          0          0      22470      22470          0   79615197          0          0
  3  101219504   79708933          0          0          0          0      46868      46868          0   79615197          0          0
  4        246          0          0          0          0          0          0          0          0          0          0          0
  5     588847        366          0          0          0          0        183        183          0          0          0          0
  6        280          0          0          0          0          0          0          0          0          0          0          0
  7     737827        382          0          0          0          0        191        191          0          0          0          0
  8        262          0          0          0          0          0          0          0          0          0          0          0
  9        221          0          0          0          0          0          0          0          0          0          0          0
 10        242          2          0          0          0          0          1          1          0          0          0          0
 11        210          0          0          0          0          0          0          0          0          0          0          0
 12        273          0          0          0          0          0          0          0          0          0          0          0
 13        193          0          0          0          0          0          0          0          0          0          0          0
 14        250          0          0          0          0          0          0          0          0          0          0          0
 15        184          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   45.816s (  698.440 blocks/s,  43.652 Mb/s)
Attachment 12: R15_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   55871232 (   17633.8 Hz)
        Other data format:  206048770 (   65032.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         56 (       0.0 Hz)
                               RESUME:         56 (       0.0 Hz)
                              SYNC100:      43803 (      13.8 Hz)
                              WR48-63:      43803 (      13.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:  205961052 (   65004.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3168.420 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2865.072
  2                0.000                0.000
  3                0.419                0.000
  4                0.000                0.000
  5                0.000             2166.184
  6                0.000                0.000
  7                0.000             1949.169
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         55          0          0          0          0          0          0          0          0          0          0          0
  1     133932         56          0          0          0          0         28         28          0          0          0          0
  2         84  103025372          0          0          0          0      18982      18982          0  102987408          0          0
  3   42389344  103017790          0          0         56         56      22017      22017          0  102973644          0          0
  4         56          0          0          0          0          0          0          0          0          0          0          0
  5    6040914       2574          0          0          0          0       1287       1287          0          0          0          0
  6        103          0          0          0          0          0          0          0          0          0          0          0
  7    7306081       2976          0          0          0          0       1488       1488          0          0          0          0
  8         78          0          0          0          0          0          0          0          0          0          0          0
  9         86          0          0          0          0          0          0          0          0          0          0          0
 10         68          0          0          0          0          0          0          0          0          0          0          0
 11         75          0          0          0          0          0          0          0          0          0          0          0
 12        116          2          0          0          0          0          1          1          0          0          0          0
 13         80          0          0          0          0          0          0          0          0          0          0          0
 14         80          0          0          0          0          0          0          0          0          0          0          0
 15         80          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.406s (  737.221 blocks/s,  46.076 Mb/s)
Attachment 13: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png
Attachment 14: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png
Attachment 15: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png
Attachment 17: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png
Attachment 18: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png
Attachment 19: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png
Attachment 20: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png
Attachment 21: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png
Attachment 22: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png
Attachment 23: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png
  343   Fri May 21 23:19:23 2021 TDSaturday 22 May 00:00-08:00
00.16 413Gb remaining /TapeData filesystem
      estimate 413/2 x 177s = 10.15h to filesystem full - OK

00.18 All histograms, statistics and merger statistics zero'd

00.57 All system wide checks OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      9      3      4      1      5      2      2      3      3      6   : 36356
aida02 :      4      8      7      2      4      3      3      3      2      3      6   : 35968
aida03 :     12      9      6      2      5      3      2      3      3      3      6   : 36824
aida04 :     11      5      9      2      2      2      2      4      3      3      6   : 37028
aida05 :     24      9      2      1      2      3      2      3      4      3      6   : 37608
aida06 :     25      6      3      4      2      4      3      3      3      3      6   : 37060
aida07 :      3      3      3      1      3      3      2      2      2      2      7   : 37044
aida08 :     20      4      2      3      1      3      2      2      3      3      6   : 36016
aida09 :     15      8      2      5      0      3      3      2      3      3      6   : 36284
aida10 :     16      6      4      1      2      3      3      2      2      4      6   : 37328
aida11 :     20      2      2      1      1      3      2      2      3      3      6   : 35936
aida12 :      1      2      1      2      3      4      1      2      4      3      6   : 36900
aida13 :     25     10      4      1      0      3      2      3      3      3      6   : 36500
aida14 :     20      6      9      2      2      5      1      2      2      4      6   : 37200
aida15 :     31      9      5      2      3      3      1      3      3      3      6   : 36500
aida16 :     24      4      2      3      1      4      2      3      3      3      6   : 36672

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

01.12 Lost activity monitor - attachment 1
      1.8.H spectra - attachments 2 & 3
      1.8.L spectra - attachments 4-6
       aida01 pulser peak width 120 ch FWHM
      Rate specttra - attachments 7 & 8
01.06 Lost activity monitor - attachment 9
      ucesb - attachment 10
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 11
      statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17
      FEE64 temperatures OK - attachment 18
      DSSSD bias & leakage currents OK - attachment 19
      Merger/Tape Server/Merger stats - attachment 20
       no merger warning/error messages since last restart
      
01.26 Analysis file S496/R14_1798 - attachment 21
       aida08 5.5% deadtime

01.17 Disk space remaining /TapeData filesystem

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  706M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.4T  395G  95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 395/2 x 177s = 9.7h to filesystem full - OK

01.30 despec shift reports no data reported by ucesb
      aida stats & tape server ok - no data being forwarded to MBS by data relay

01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811

04.35 System wide checks all OK *except*

Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  661M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  337G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 337/2 x 177s = 8.3h to filesystem full - OK

FEE64 temperatures OK - attachment 22
ADC data item statistics OK - attachment 24
DSSSD bias & leakage current OK - attachment 26
analysis of file S496/R14_1920 - attachment 28
 aida04 deadtime 5.0%

07.10 Ssystem wide checks all OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  662M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  287G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 287/2 x 178s = 7.1h to filesystem full - OK

FEE64 temperatures OK - attachment 23
ADC data item statistics OK - attachment 25
DSSSD bias & leakage current OK - attachment 27
analysis of file S496/R14_1920 - attachment 29
 aida04 deadtime 4.4%

07.35 Found MBS data monitor kill'd
      Restarted MBS data relay and data monitor OK

08.00 beam off - end of S496

08.28 Tape Server -> no storage mode
      DAQ continues

09.58 All slow comparators -> 0x64
      BNC PB-5 OFF

      file S496/R15

      alpha background

      Tape server 200kb/s, merger 260k data items/s
Attachment 1: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png
Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png
Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png
Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png
Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png
Attachment 9: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png
Attachment 10: Screenshot_2021-05-22_ucesb.png
Screenshot_2021-05-22_ucesb.png
Attachment 11: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png
Attachment 12: Screenshot_2021-05-22_Statistics_aidas-gsi(5).png
Screenshot_2021-05-22_Statistics_aidas-gsi(5).png
Attachment 13: Screenshot_2021-05-22_Statistics_aidas-gsi(4).png
Screenshot_2021-05-22_Statistics_aidas-gsi(4).png
Attachment 14: Screenshot_2021-05-22_Statistics_aidas-gsi(3).png
Screenshot_2021-05-22_Statistics_aidas-gsi(3).png
Attachment 15: Screenshot_2021-05-22_Statistics_aidas-gsi(2).png
Screenshot_2021-05-22_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-22_Statistics_aidas-gsi(1).png
Screenshot_2021-05-22_Statistics_aidas-gsi(1).png
Attachment 17: Screenshot_2021-05-22_Statistics_aidas-gsi.png
Screenshot_2021-05-22_Statistics_aidas-gsi.png
Attachment 18: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 19: 800.png
800.png
Attachment 20: 801.png
801.png
Attachment 21: R14_1798
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254662088 ( 1433466.5 Hz)
        Other data format:    7257912 (   40854.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        256 (       1.4 Hz)
                               RESUME:        257 (       1.4 Hz)
                              SYNC100:      32661 (     183.8 Hz)
                              WR48-63:      32661 (     183.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7192077 (   40483.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     365795 (    2059.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      177.655 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.112                0.000
  1                5.898                0.000
  2                0.060                0.000
  3                7.669                0.000
  4                0.073                7.902
  5                7.648                0.000
  6                0.356                0.000
  7                9.852                0.000
  8                1.513                0.000
  9                0.000               15.513
 10                2.639                0.000
 11                0.903                0.000
 12                0.687                0.000
 13                0.000                6.517
 14                0.073                0.000
 15                0.212                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5664289       2546          0          0          5          5        737        737          0       1062          0      19821
  1   24002581       6376          0          0         39         39       3149       3149          0          0          0      40581
  2   12140787    1878978          0          0          3          3       1758       1758          0    1875456          0      32794
  3   29094115    1803103          0          0         54         54       3917       3917          0    1795161          0      46229
  4    3034023    1008057          0          0          6          6        480        480          0    1007085          0       8526
  5   29014047     561174          0          0         54         54       3636       3636          0     553794          0      24714
  6   11191428    1180757          0          0          3          3       1528       1528          0    1177695          0      17521
  7   34103710     790567          0          0         51         52       4320       4320          0     781824          0      29424
  8   17182670       4348          0          0          9          9       2165       2165          0          0          0      41538
  9    2132311        520          0          0          0          0        260        260          0          0          0       3304
 10   25432181       6308          0          0         12         12       3142       3142          0          0          0      25046
 11   16063496       3832          0          0          8          8       1908       1908          0          0          0       7738
 12   11072472       2756          0          0          4          4       1374       1374          0          0          0      35884
 13    2789016        664          0          0          0          0        332        332          0          0          0       2683
 14   11165568       2934          0          0          4          4       1463       1463          0          0          0      24254
 15   20579394       4992          0          0          4          4       2492       2492          0          0          0       5738
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.474s (  771.565 blocks/s,  48.223 Mb/s)
Attachment 22: Screenshot_2021-05-22_Statistics_aidas-gsi(7).png
Screenshot_2021-05-22_Statistics_aidas-gsi(7).png
Attachment 23: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 24: Screenshot_2021-05-22_Statistics_aidas-gsi(6).png
Screenshot_2021-05-22_Statistics_aidas-gsi(6).png
Attachment 25: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 26: 802.png
802.png
Attachment 27: 804.png
804.png
Attachment 28: R14_1867
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254778017 ( 1461534.6 Hz)
        Other data format:    7141983 (   40970.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        228 (       1.3 Hz)
                               RESUME:        228 (       1.3 Hz)
                              SYNC100:      32649 (     187.3 Hz)
                              WR48-63:      32649 (     187.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7076229 (   40592.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     358653 (    2057.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      174.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.023                0.000
  1                8.071                0.000
  2                0.956                0.000
  3                6.628                0.000
  4                0.186               12.353
  5                5.668                0.000
  6                0.255                0.000
  7                8.669                0.000
  8                1.570                0.000
  9                0.000               28.420
 10                2.125                0.000
 11                0.164                0.000
 12                0.103                0.000
 13                0.000               11.688
 14                0.094                0.000
 15                1.340                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5294176       2413          0          0          1          1        685        685          0       1041          0      18715
  1   22878067       6136          0          0         44         44       3024       3024          0          0          0      38998
  2   11959613    1840531          0          0          7          7       1681       1681          0    1837155          0      31909
  3   28914032    1781740          0          0         51         51       3845       3845          0    1773948          0      44969
  4    2970776     981532          0          0          7          7        454        454          0     980610          0       8409
  5   29112496     551278          0          0         40         40       3688       3688          0     543822          0      24330
  6   11438278    1149690          0          0          4          4       1643       1643          0    1146396          0      17361
  7   34470364     802023          0          0         41         41       4342       4342          0     793257          0      30254
  8   17110979       4312          0          0          6          6       2150       2150          0          0          0      40990
  9    2129568        524          0          0          0          0        262        262          0          0          0       3068
 10   25946722       6422          0          0          9          9       3202       3202          0          0          0      25071
 11   16268948       3960          0          0          3          3       1977       1977          0          0          0       7119
 12   11320269       2746          0          0          3          3       1370       1370          0          0          0      35684
 13    2799924        716          0          0          0          0        358        358          0          0          0       2469
 14   11389376       2772          0          0          2          2       1384       1384          0          0          0      23994
 15   20774429       5188          0          0         10         10       2584       2584          0          0          0       5313
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.000s (  744.186 blocks/s,  46.512 Mb/s)
Attachment 29: R14_1920
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254781729 ( 1428365.9 Hz)
        Other data format:    7138271 (   40018.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        226 (       1.3 Hz)
                               RESUME:        227 (       1.3 Hz)
                              SYNC100:      32665 (     183.1 Hz)
                              WR48-63:      32665 (     183.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7072488 (   39650.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     352885 (    1978.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      178.373 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.017                2.836
  1                6.610                0.000
  2                0.573                0.000
  3                7.826                0.000
  4                0.024               14.831
  5                7.215                0.000
  6                0.482                0.000
  7                6.876                0.000
  8                0.394                0.000
  9                0.000               17.832
 10                1.157                0.000
 11                1.395                0.000
 12                0.037                0.000
 13                0.004                9.016
 14                0.101                0.000
 15                1.578                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5306415       2397          0          0          1          1        665        665          0       1065          0      18998
  1   22746335       6008          0          0         36         36       2968       2968          0          0          0      37812
  2   11931383    1875609          0          0          5          5       1756       1756          0    1872087          0      31542
  3   28584441    1795513          0          0         57         57       3773       3773          0    1787853          0      43417
  4    2954259     971818          0          0          3          3        494        494          0     970824          0       8522
  5   28608195     543853          0          0         51         52       3567       3567          0     536616          0      24072
  6   11355091    1133078          0          0          6          6       1573       1573          0    1129920          0      16979
  7   34582740     783003          0          0         34         34       4406       4406          0     774123          0      29465
  8   17425265       4332          0          0          7          7       2159       2159          0          0          0      40811
  9    2130016        550          0          0          0          0        275        275          0          0          0       3176
 10   26903024       6616          0          0          4          4       3304       3304          0          0          0      24219
 11   16400506       4036          0          0          5          5       2013       2013          0          0          0       7163
 12   11085724       2934          0          0          1          1       1466       1466          0          0          0      35361
 13    2664218        650          0          0          1          1        324        324          0          0          0       2574
 14   11252748       2738          0          0          4          4       1365       1365          0          0          0      23421
 15   20851369       5136          0          0         11         11       2557       2557          0          0          0       5353
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.672s (  732.737 blocks/s,  45.796 Mb/s)
  342   Fri May 21 15:03:48 2021 CAFriday 21st May 16:00 - 00:00 shift
16:00 CA takes over
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

16:14 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x570 : 	 112  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

16:17 temperatures ok - attachment 1
      statistics ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      analysis of R14_1625 - FEE7 deadtime at 4.5%

18:06 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x571 : 	 113  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:07 temperatures ok - attachment 4
      statistics ok - attachment 5
      detector bias / leakage currents ok - attachment 6
      analysis of R14_1662 - FEE7 deadtime at 4.3%

19:07 DAQ continues ok - writing to file R14_1682

20:07 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

20:10 temperatures ok - attachment 7
      statistics ok - attachment 8
      detector bias / leakage currents ok - attachment 9
      analysis of R14_1703 - max deadtime at 4.0% - attachment 10

20:35 rate spectra - attachment 11
      HEC spectra - attachment 12 & 13

20:36 all histograms zeroed

20:37 DAQ continues ok - writing to file R14_1712

20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.

      ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files

21:12 no beam - writing to file R14_1723

21:17 beam is back

22:07 system wide checks all ok - WR decoder fault same as entry at 20:07

22:08 temperatures ok - attachment 14
      statistics ok - attachment 15
      detector bias / leakage currents ok - attachment 16
      analysis of R14_1740 - max deadtime at 4.2% - attachment 17








   
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png
Screenshot_from_2021-05-21_15-14-59.png
Attachment 2: Screenshot_from_2021-05-21_15-15-38.png
Screenshot_from_2021-05-21_15-15-38.png
Attachment 3: Screenshot_from_2021-05-21_15-15-54.png
Screenshot_from_2021-05-21_15-15-54.png
Attachment 4: Screenshot_from_2021-05-21_17-06-56.png
Screenshot_from_2021-05-21_17-06-56.png
Attachment 5: Screenshot_from_2021-05-21_17-07-18.png
Screenshot_from_2021-05-21_17-07-18.png
Attachment 6: Screenshot_from_2021-05-21_17-07-47.png
Screenshot_from_2021-05-21_17-07-47.png
Attachment 7: Screenshot_from_2021-05-21_19-09-19.png
Screenshot_from_2021-05-21_19-09-19.png
Attachment 8: Screenshot_from_2021-05-21_19-09-59.png
Screenshot_from_2021-05-21_19-09-59.png
Attachment 9: Screenshot_from_2021-05-21_19-10-15.png
Screenshot_from_2021-05-21_19-10-15.png
Attachment 10: Screenshot_from_2021-05-21_19-14-45.png
Screenshot_from_2021-05-21_19-14-45.png
Attachment 11: Screenshot_from_2021-05-21_19-25-51.png
Screenshot_from_2021-05-21_19-25-51.png
Attachment 12: Screenshot_from_2021-05-21_19-30-14.png
Screenshot_from_2021-05-21_19-30-14.png
Attachment 13: Screenshot_from_2021-05-21_19-32-13.png
Screenshot_from_2021-05-21_19-32-13.png
Attachment 14: Screenshot_from_2021-05-21_21-08-28.png
Screenshot_from_2021-05-21_21-08-28.png
Attachment 15: Screenshot_from_2021-05-21_21-08-51.png
Screenshot_from_2021-05-21_21-08-51.png
Attachment 16: Screenshot_from_2021-05-21_21-09-06.png
Screenshot_from_2021-05-21_21-09-06.png
Attachment 17: Screenshot_from_2021-05-21_21-12-03.png
Screenshot_from_2021-05-21_21-12-03.png
  341   Fri May 21 07:03:50 2021 OH, TDFriday 21 May 08:00-16:00
08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png
210521_0829_Stats.png
Attachment 2: 210521_0829_Temp.png
210521_0829_Temp.png
Attachment 3: 210521_0830_Bias.png
210521_0830_Bias.png
Attachment 4: 210521_0957.png
210521_0957.png
Attachment 5: 210521_0957_Temp.png
210521_0957_Temp.png
Attachment 6: 210521_0958_Bias.png
210521_0958_Bias.png
Attachment 7: 210521_1232_Stats.png
210521_1232_Stats.png
Attachment 8: 210521_1233_Bias.png
210521_1233_Bias.png
Attachment 9: 210521_1233_Temp.png
210521_1233_Temp.png
  340   Fri May 21 06:09:28 2021 MLsystem wide checks

System  wide checks at 7am CET

All seems fine, nothing significant to report

Slight increase of dead time noted in FEE 3, 5 and 7.

-----
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


-----

White Rabbit:

         Base         Current     Difference
aida05 fault      0x500 :      0x56e :      110  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     23      5      6      5      0      4      1      3      3      3      6   : 36484
aida02 :     10      3      3      2      3      2      2      2      3      3      6   : 35952
aida03 :      2      5      1      4      5      2      2      3      3      3      6   : 36608
aida04 :      1      3      4      0      4      3      2      3      3      3      6   : 36572
aida05 :     19      9      3      2      2      3      1      3      4      3      6   : 37380
aida06 :     13     11      1      5      3      4      4      2      3      3      6   : 36860
aida07 :     12      2      4      3      3      2      1      3      2      2      7   : 37280
aida08 :     25      5      7      1      1      3      3      3      3      3      6   : 36828
aida09 :     22      5      6      4      1      3      2      3      3      3      6   : 36640
aida10 :     23      8      6      3      2      3      2      2      2      4      6   : 37212
aida11 :     24     14      7      3      3      2      3      3      3      3      6   : 36960
aida12 :     16     10      5      3      1      2      1      2      4      3      6   : 36736
aida13 :     23      6      4      2      2      3      2      3      3      3      6   : 36620
aida14 :     22      7      8      4      3      4      1      2      2      4      6   : 37200
aida15 :     10     12      9      2      1      3      1      3      3      3      6   : 36376
aida16 :     18      5      6      2      0      3      2      4      3      3      6   : 37008

 

Attachment 1: 2021-05-21_7h00_LeakCurrent.png
2021-05-21_7h00_LeakCurrent.png
Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png
2021-05-21_7h00_Stats_GoodEvents.png
Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png
2021-05-21_7h00_Stats_ADCitems.png
Attachment 4: 2021-05-21_7h00_stats_Correlation.png
2021-05-21_7h00_stats_Correlation.png
Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png
2021-05-21_7h00_Stats_PAUSE.png
Attachment 6: 2021-05-21_7h00_Stats_RESUME.png
2021-05-21_7h00_Stats_RESUME.png
Attachment 7: 2021-05-21_7h00_temperature.png
2021-05-21_7h00_temperature.png
Attachment 8: 2021-05-21_7h00_ucesb.png
2021-05-21_7h00_ucesb.png
  339   Fri May 21 02:17:47 2021 MLsystem wide checks

Wide check at 3am -

All fine.

Same output as at the beginning of the shift (see my previous entry) and attachments.

 

Attachment 1: 2021-05-21_3h05_Leakcurrent.png
2021-05-21_3h05_Leakcurrent.png
Attachment 2: 2021-05-21_3h05_stats_adcItem.png
2021-05-21_3h05_stats_adcItem.png
Attachment 3: 2021-05-21_3h05_stats_Correlation.png
2021-05-21_3h05_stats_Correlation.png
Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png
2021-05-21_3h05_stats_GoodEvents.png
Attachment 5: 2021-05-21_3h05_stats_PAUSE.png
2021-05-21_3h05_stats_PAUSE.png
Attachment 6: 2021-05-21_3h05_stats_RESUME.png
2021-05-21_3h05_stats_RESUME.png
Attachment 7: 2021-05-21_3h05_temperature.png
2021-05-21_3h05_temperature.png
Attachment 8: 2021-05-21_3h05_ucesb.png
2021-05-21_3h05_ucesb.png
Attachment 9: 2021-05-21_3h05_aidaGrafana.png
2021-05-21_3h05_aidaGrafana.png
  338   Thu May 20 23:26:11 2021 Marc Labiche (ML)system wide checks

System Wide check at begining of the night shift:

-----

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

-----

White rabbit status:

         Base         Current     Difference
aida05 fault      0x500 :      0x56d :      109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Memory info from FEE cards

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      7      7      6      1      5      2      2      3      3      6   : 36472
aida02 :     19      5      4      4      4      3      3      3      2      3      6   : 36020
aida03 :     22     10      2      2      5      2      2      3      3      3      6   : 36680
aida04 :     12      7      1      0      4      3      1      4      3      3      6   : 36856
aida05 :     16      7      5      3      1      3      2      3      3      3      6   : 36584
aida06 :     20      7      5      6      2      4      4      2      3      3      6   : 36888
aida07 :     13      7      5      2      3      2      1      3      2      2      7   : 37308
aida08 :     18      7      5      3      0      2      3      3      3      3      6   : 36656
aida09 :     21      6      4      4      1      4      2      3      3      3      6   : 36740
aida10 :     21     10      2      3      2      3      2      2      2      4      6   : 37156
aida11 :     23      2      4      4      3      2      2      3      2      4      6   : 37612
aida12 :     24      1      4      3      2      2      1      2      4      3      6   : 36744
aida13 :     26      9      4      2      1      4      3      2      3      3      6   : 36464
aida14 :     19      9      9      2      3      4      1      2      2      4      6   : 37156
aida15 :     10      7     10      0      1      3      1      3      3      3      6   : 36288
aida16 :     28      7      3      2      1      3      3      3      3      3      6   : 36824

Attachment 1: 2021-05-21_0h20_ucesb.png
2021-05-21_0h20_ucesb.png
Attachment 2: 2021-05-21_0h10_Leakcurrent.png
2021-05-21_0h10_Leakcurrent.png
Attachment 3: 2021-05-21_0h10_Stat_adcItem.png
2021-05-21_0h10_Stat_adcItem.png
Attachment 4: 2021-05-21_0h10_Stats_Correlation.png
2021-05-21_0h10_Stats_Correlation.png
Attachment 5: 2021-05-21_0h10_Stats_GoodEvents.png
2021-05-21_0h10_Stats_GoodEvents.png
Attachment 6: 2021-05-21_0h10_Stats_PAUSE.png
2021-05-21_0h10_Stats_PAUSE.png
Attachment 7: 2021-05-21_0h10_stats_RESUME.png
2021-05-21_0h10_stats_RESUME.png
Attachment 8: 2021-05-21_0h10_temperature.png
2021-05-21_0h10_temperature.png
Attachment 9: 2021-05-21_0h30_AIDAgrafana.png
2021-05-21_0h30_AIDAgrafana.png
  337   Thu May 20 14:56:33 2021 OHThursday 20th May 16:00 - 24:00
16:00 p+n threshold 0xc (120keV)
      n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4
      n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4

      Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure.

17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector

18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now.
      The deadtimes in FEE6 and 8 have risen to around 6.25%.
      Looking at this mornings and yesterday evenings fill times this behavior isn't observed

18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago

18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2
      This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file)

18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8

18:15 System wide checks ok except WR as usual
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56d : 	 109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics increased at 18:02 still working on - attachment 2
      FEE temp - attachment 3
      Bias and leakage current - attachment 4


20:51 System wide checks ok - WR same as before
      Statistics still high - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7 

22:52 Statistics still high - attachment 8
      Temp ok - attachment 9
      Bias and leakage current ok - attachment 10
      System wide checks ok - WR difference still 109
Attachment 1: 210520_18002_MBSRate.png
210520_18002_MBSRate.png
Attachment 2: 210520_1916_Stats.png
210520_1916_Stats.png
Attachment 3: 210520_1916_Temp.png
210520_1916_Temp.png
Attachment 4: 210520_1917_Bias.png
210520_1917_Bias.png
Attachment 5: 21050_2052_Stats.png
21050_2052_Stats.png
Attachment 6: 210520_2052_Temp.png
210520_2052_Temp.png
Attachment 7: 210520_2053_Bias.png
210520_2053_Bias.png
Attachment 8: 210520_2251_Stats.png
210520_2251_Stats.png
Attachment 9: 210520_2251_Temp.png
210520_2251_Temp.png
Attachment 10: 210520_2254_Bias.png
210520_2254_Bias.png
  336   Thu May 20 07:03:02 2021 CAThursday May 20th 08:00 - 16:00 shift
08:00 CA takes over

08:04 DAQ continues ok - writing to file R14_989

09:30 all system wide checks ok *except* WR decoder status;

      
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56a : 	 106  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

09:34 temperatures ok - attachment 1
      stats ok - attachment 2
      bias/leakage currents ok - attachment 3
      R14_1016 analysis - max deadtime ~4.3% - attachment 4

11:30 all system wide checks ok *except* WR decoder status;

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56b : 	 107  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

11:34 temperatures ok - attachment 5
      stats ok - attachment 6
      bias/leakage currents ok - attachment 7
      R14_1052 analysis - max deadtime ~3.6% - attachment 8

13:05 beam optimisation currently taking place
      DAQ continues ok and writing to file R14_1085

13:17 no beam

13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.

13:31 temperatures ok - attachment 9
      stats ok - attachment 10
      bias/leakage currents ok - attachment 11
      R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12

13:37 beam is back - rate spectra - attachment 13
      DAQ ok - writing to file R14_1088

13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14

14:21 beam off - issue with UNILAC
      DESPEC also report issues with their DAQ
      AIDA continues to forward data to MBS ok - lower rates while beam off

14.30 1.8.H spectra - attachments 16-18
      1.8.L spectra - attachments 19-22
       aida01 pulser peak width 99 ch FWHM
      Rate spectra - attachment 23
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
      DSSSD bias & leakage currents - attachment 25
      Merger/Tape Server/Merger stats - attachment 26



14.41 All histograms & stats zero'd

14.55 Lost Activity Monitors - attachments 27, 28 & 35
      Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler

15:00 beam is back

15:05 AIDA writing to file R14_1108
      ADC data items - attachment 36

15:27 system wide checks ok *except* WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56c : 	 108  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

15:30 temperatures ok - attachment 37
      stats ok - attachment 38
      bias/leakage currents ok - attachment 39
      R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40
Attachment 1: Screenshot_from_2021-05-20_08-33-49.png
Screenshot_from_2021-05-20_08-33-49.png
Attachment 2: Screenshot_from_2021-05-20_08-34-10.png
Screenshot_from_2021-05-20_08-34-10.png
Attachment 3: Screenshot_from_2021-05-20_08-34-44.png
Screenshot_from_2021-05-20_08-34-44.png
Attachment 4: Screenshot_from_2021-05-20_08-37-13.png
Screenshot_from_2021-05-20_08-37-13.png
Attachment 5: Screenshot_from_2021-05-20_10-32-22.png
Screenshot_from_2021-05-20_10-32-22.png
Attachment 6: Screenshot_from_2021-05-20_10-32-47.png
Screenshot_from_2021-05-20_10-32-47.png
Attachment 7: Screenshot_from_2021-05-20_10-33-04.png
Screenshot_from_2021-05-20_10-33-04.png
Attachment 8: Screenshot_from_2021-05-20_10-36-31.png
Screenshot_from_2021-05-20_10-36-31.png
Attachment 9: Screenshot_from_2021-05-20_12-29-26.png
Screenshot_from_2021-05-20_12-29-26.png
Attachment 10: Screenshot_from_2021-05-20_12-29-49.png
Screenshot_from_2021-05-20_12-29-49.png
Attachment 11: Screenshot_from_2021-05-20_12-30-20.png
Screenshot_from_2021-05-20_12-30-20.png
Attachment 12: Screenshot_from_2021-05-20_12-33-48.png
Screenshot_from_2021-05-20_12-33-48.png
Attachment 13: Screenshot_from_2021-05-20_12-38-42.png
Screenshot_from_2021-05-20_12-38-42.png
Attachment 14: Screenshot_from_2021-05-20_12-38-42.png
Screenshot_from_2021-05-20_12-38-42.png
Attachment 15: Screenshot_from_2021-05-20_12-43-18.png
Screenshot_from_2021-05-20_12-43-18.png
Attachment 16: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(7).png
Attachment 17: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(6).png
Attachment 18: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(5).png
Attachment 19: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(4).png
Attachment 20: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(3).png
Attachment 21: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(2).png
Attachment 22: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(1).png
Attachment 23: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi.png
Attachment 24: Screenshot_2021-05-20_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-20_AIDA_Alerting_-_Grafana.png
Attachment 25: 710.png
710.png
Attachment 26: 711.png
711.png
Attachment 27: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(2).png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(2).png
Attachment 28: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(1).png
Attachment 29: Screenshot_2021-05-20_Statistics_aidas-gsi(5).png
Screenshot_2021-05-20_Statistics_aidas-gsi(5).png
Attachment 30: Screenshot_2021-05-20_Statistics_aidas-gsi(4).png
Screenshot_2021-05-20_Statistics_aidas-gsi(4).png
Attachment 31: Screenshot_2021-05-20_Statistics_aidas-gsi(3).png
Screenshot_2021-05-20_Statistics_aidas-gsi(3).png
Attachment 32: Screenshot_2021-05-20_Statistics_aidas-gsi(2).png
Screenshot_2021-05-20_Statistics_aidas-gsi(2).png
Attachment 33: Screenshot_2021-05-20_Statistics_aidas-gsi(1).png
Screenshot_2021-05-20_Statistics_aidas-gsi(1).png
Attachment 34: Screenshot_2021-05-20_Statistics_aidas-gsi.png
Screenshot_2021-05-20_Statistics_aidas-gsi.png
Attachment 35: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi.png
Attachment 36: Screenshot_from_2021-05-20_14-07-22.png
Screenshot_from_2021-05-20_14-07-22.png
Attachment 37: Screenshot_from_2021-05-20_14-28-33.png
Screenshot_from_2021-05-20_14-28-33.png
Attachment 38: Screenshot_from_2021-05-20_14-29-04.png
Screenshot_from_2021-05-20_14-29-04.png
Attachment 39: Screenshot_from_2021-05-20_14-29-18.png
Screenshot_from_2021-05-20_14-29-18.png
Attachment 40: Screenshot_from_2021-05-20_14-31-50.png
Screenshot_from_2021-05-20_14-31-50.png
  335   Thu May 20 01:00:30 2021 MuneerahThu May 20 00:00-08:00

02:00 System check

The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!

The rates on ucesb got back to work by around 01:12

Deat Time check ok R14_870, attachment 1

Statistics are ok attachment 2

Temperatures are ok attacment 3

Voltages are ok attachment 4

 

system wide check

All ok except

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White rabbite

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      8      8      5      0      6      1      2      2      2      7   : 37296
aida02 :     19      8     11      2      3      4      2      2      3      3      6   : 36412
aida03 :     14      6      3      4      1      3      1      3      3      3      6   : 36312
aida04 :     25     12     12      4      3      4      1      4      3      3      6   : 37316
aida05 :      4      8      3      4      1      2      2      4      3      3      6   : 36928
aida06 :     42     12      8      7      2      3      3      4      3      3      6   : 37736
aida07 :     24      6      7      3      3      3      1      3      2      2      7   : 37536
aida08 :     10     15      7      2      1      4      2      3      3      3      6   : 36752
aida09 :     29     13      6      6      4      3      3      3      3      3      6   : 37244
aida10 :     18     12      7      2      2      3      3      2      2      4      6   : 37464
aida11 :     20      8      0      5      5      4      2      3      3      3      6   : 36976
aida12 :      7      2      4      3      2      3      2      2      4      3      6   : 37068
aida13 :      2      5      5      4      2      3      2      3      3      3      6   : 36608
aida14 :      9     13      8      2      2      2      3      3      2      4      6   : 37836
aida15 :     24     12     12      1      0      3      2      3      3      3      6   : 36640
aida16 :     12      9      9      2      1      3      2      4      3      3      6   : 37128

6:45 System check

Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9

System wide check are all ok except:

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit 1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20     11      3      5      0      6      1      2      3      2      7   : 38264
aida02 :     14      9      9      2      3      3      3      2      3      3      6   : 36496
aida03 :     13      7      3      5      2      3      2      3      3      3      6   : 36668
aida04 :     30     15     13      4      3      4      1      3      4      3      6   : 37888
aida05 :     17      9      3      3      1      2      2      4      3      3      6   : 36956
aida06 :     40     16      5      6      2      3      2      3      2      4      6   : 37936
aida07 :      3      2      3      2      3      4      2      2      2      2      7   : 37196
aida08 :     17     10      6      2      1      3      3      3      3      3      6   : 36852
aida09 :     29     11      7      6      3      3      2      4      3      3      6   : 37436
aida10 :      1     10      5      1      1      3      3      2      2      4      6   : 37252
aida11 :     30      9      0      5      4      4      2      3      3      3      6   : 36960
aida12 :     19     10      5      3      2      3      2      2      4      3      6   : 37196
aida13 :     22      9      6      3      2      3      3      3      3      3      6   : 36960
aida14 :     19      6      9      2      2      3      3      2      2      4      6   : 37452
aida15 :      6      8     11      2      1      3      2      3      3      3      6   : 36616
aida16 :     19     14      5      3      1      3      1      4      3      3      6   : 36908

 

07:33 system check

Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13

System wide check

all ok except

ADC all failed

White rabbit

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      9      4      6      0      5      1      2      2      2      7   : 37152
aida02 :     19      6     13      1      3      3      4      2      3      3      6   : 36780
aida03 :     23      5      6      4      2      3      1      3      3      3      6   : 36452
aida04 :     24     11     14      3      3      4      2      3      3      3      6   : 37048
aida05 :     17      3      6      3      1      2      2      3      3      3      6   : 36444
aida06 :     36     15      4      9      2      4      2      4      3      3      6   : 37608
aida07 :     20      8      5      3      3      4      2      2      2      2      7   : 37376
aida08 :     21     11     10      2      1      4      2      2      2      4      6   : 37324
aida09 :     27     14      7      6      3      3      2      4      3      3      6   : 37452
aida10 :     25      9      5      2      2      3      3      2      2      4      6   : 37436
aida11 :     30     14      2      4      5      3      2      3      3      3      6   : 36936
aida12 :     22      3      5      3      2      3      2      3      3      3      6   : 36640
aida13 :     10      7      1      4      2      3      3      2      3      3      6   : 36336
aida14 :     13      9     12      2      3      3      2      3      2      4      6   : 37820
aida15 :     20      8     11      3      0      3      2      3      3      3      6   : 36640
aida16 :     23      6      6      2      0      2      1      3      4      3      6   : 37164

 

 

 

 

 

Attachment 1: 210520_0156_DT.png
210520_0156_DT.png
Attachment 2: 210520_0157_Stats.png
210520_0157_Stats.png
Attachment 3: 210520_0157_Temps.png
210520_0157_Temps.png
Attachment 4: 210520_0157_Voltages.png
210520_0157_Voltages.png
Attachment 5: 210520_0537_Temps.png
210520_0537_Temps.png
Attachment 6: 210520_0539_VOltages.png
210520_0539_VOltages.png
Attachment 7: 210520_0540_Histo.png
210520_0540_Histo.png
Attachment 8: 210520_0541_Rates.png
210520_0541_Rates.png
Attachment 9: 210520_0542_DT.png
210520_0542_DT.png
Attachment 10: 210520_0730_DT.png
210520_0730_DT.png
Attachment 11: 210520_0730_Voltages.png
210520_0730_Voltages.png
Attachment 12: 210520_0731_Stats.png
210520_0731_Stats.png
Attachment 13: 210520_0731_Tems.png
210520_0731_Tems.png
  334   Wed May 19 20:02:27 2021 NHAIDA Dead Time Analysis
Some plots of AIDA dead time (From PAUSE/RESUME data items) against time.

The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead.

Beam spills are indicated by the red bars (1 = on spill)

Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown
Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates .

The much quieter p+n strips show a similar pattern but much less frequently.

Deadtime does not always sync up between FEE64s it seems.

IN all case the pause/resume interval seems to only last for less than 100 ms.
Attachment 1: dt_fee2.png
dt_fee2.png
Attachment 2: dt_fee4.png
dt_fee4.png
Attachment 3: dt_fee6.png
dt_fee6.png
Attachment 4: dt_fee8.png
dt_fee8.png
Attachment 5: dt_fee9.png
dt_fee9.png
Attachment 6: dt_fee9_lr.png
dt_fee9_lr.png
  333   Wed May 19 18:50:01 2021 LSWednesday 19th May 20:00 to 24:00
20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)

20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)

21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.8M items/s
Tape server ~10MB/s

Analysed R14_781 deadtimes are okay (Fig 9)

21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)

22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)

22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)

23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~3M items/s
Tape server ~10MB/s

Analysed R14_819 deadtimes are okay (Fig 19)

23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21)
Attachment 1: 210519_Stats_2000.png
210519_Stats_2000.png
Attachment 2: Deadtimes_R14_762.png
Deadtimes_R14_762.png
Attachment 3: 210519_Stats_2030.png
210519_Stats_2030.png
Attachment 4: Deadtimes_R14_771.png
Deadtimes_R14_771.png
Attachment 5: 210519_Stats_2100.png
210519_Stats_2100.png
Attachment 6: 210519_FEEtemps_2100.png
210519_FEEtemps_2100.png
Attachment 7: 210519_BIAS_2100.png
210519_BIAS_2100.png
Attachment 8: 210519_Spectra_2100.png
210519_Spectra_2100.png
Attachment 9: deadtimes_R14_781.png
deadtimes_R14_781.png
Attachment 10: 210519_Stats_2130.png
210519_Stats_2130.png
Attachment 11: deadtimes_R14_789.png
deadtimes_R14_789.png
Attachment 12: 210519_Stats_2200.png
210519_Stats_2200.png
Attachment 13: deadtimes_R14_799.png
deadtimes_R14_799.png
Attachment 14: 210519_Stats_2230.png
210519_Stats_2230.png
Attachment 15: deatimes_R14_807.png
deatimes_R14_807.png
Attachment 16: 210519_Stats_2300.png
210519_Stats_2300.png
Attachment 17: 210519_FEEtemps_2300.png
210519_FEEtemps_2300.png
Attachment 18: 210519_bias_2300.png
210519_bias_2300.png
Attachment 19: deatimes_R14_819.png
deatimes_R14_819.png
Attachment 20: 210519_Stats_2330.png
210519_Stats_2330.png
Attachment 21: deadtimes_R14_827.png
deadtimes_R14_827.png
  332   Wed May 19 14:54:36 2021 CBWednesday 19 May 16:00 - 20:00
15:55 Took over from OH

16:00
Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System wide checks
All OK, except ADC all fail (as before) and WR

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_691. Attach 4. Dead time <5%

16:34 MBS changes file.


17:00
Stats OK - attach 5
Temps OK - as before
System wide checks as before
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


17:55
Stats OK - attach 6
Temps OK - as before

Leakage current approaching 10 uA - attach 7

System-wide checks - as before except
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x568 : 	 104  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



18:01 MBS switches to new file



18:52
Stats OK - attach 8
Temps OK - as before

System-wide checks as before

Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9

19:36 Stats OK - attach 10
Bias - attach 11

19:48 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_691
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251761346 (  995725.1 Hz)
        Other data format:   10158654 (   40177.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.8 Hz)
                               RESUME:        199 (       0.8 Hz)
                              SYNC100:      32942 (     130.3 Hz)
                              WR48-63:      32942 (     130.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10092372 (   39915.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     503210 (    1990.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      252.842 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.053               17.526
  1                1.803                0.000
  2                0.604                2.918
  3                3.773                0.000
  4                0.070               17.512
  5                9.601                0.000
  6                0.278                0.000
  7                9.149                0.000
  8                0.114                0.000
  9                0.000               64.964
 10                0.536                0.000
 11                0.070                0.000
 12                0.024                6.415
 13                0.000               66.391
 14                0.036                7.395
 15                0.097                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6023981       3019          0          0          3          3        749        749          0       1515          0      26768
  1   22620294       5924          0          0         14         14       2948       2948          0          0          0      55394
  2   11082857    2607083          0          0          5          5       1799       1799          0    2603475          0      43681
  3   30564809    2580290          0          0         26         26       4295       4295          0    2571648          0      65104
  4    3958862    1393424          0          0          7          7        678        678          0    1392054          0      11940
  5   37388332     780379          0          0         62         62       4752       4752          0     770751          0      33878
  6   10191568    1631763          0          0          8          8       1480       1480          0    1628787          0      23955
  7   39960208    1134610          0          0         55         55       5179       5179          0    1124142          0      42303
  8   17144158       4230          0          0          3          3       2112       2112          0          0          0      56568
  9    2458065        662          0          0          0          0        331        331          0          0          0       4321
 10   20331742       4810          0          0          6          6       2399       2399          0          0          0      34970
 11   11809904       2968          0          0          4          4       1480       1480          0          0          0      10406
 12    9134013       2250          0          0          1          1       1124       1124          0          0          0      49240
 13    2508032        620          0          0          0          0        310        310          0          0          0       3587
 14    8541585       2182          0          0          2          2       1089       1089          0          0          0      33394
 15   18042936       4440          0          0          3          3       2217       2217          0          0          0       7701
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.672s (  950.348 blocks/s,  59.397 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: Analysis_R14_741
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253670855 ( 1280798.3 Hz)
        Other data format:    8249145 (   41650.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        236 (       1.2 Hz)
                               RESUME:        236 (       1.2 Hz)
                              SYNC100:      32737 (     165.3 Hz)
                              WR48-63:      32737 (     165.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8183199 (   41317.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     413031 (    2085.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      198.057 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                2.771
  1                4.649                0.000
  2                0.028                0.000
  3                4.808                0.000
  4                0.111                7.488
  5                7.093                0.000
  6                0.128                0.000
  7               10.340                0.000
  8                0.971                0.000
  9                0.000               24.260
 10                0.713                0.000
 11                0.245                0.000
 12                0.122                0.000
 13                0.000               21.524
 14                0.232                0.000
 15                1.605                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5709345       2725          0          0          1          1        769        769          0       1185          0      22625
  1   23799538       6034          0          0         27         27       2990       2990          0          0          0      44162
  2   11722022    2092139          0          0          1          1       1767       1767          0    2088603          0      36240
  3   29654318    2044332          0          0         38         38       3940       3940          0    2036376          0      53125
  4    3438629    1158241          0          0          8          8        579        579          0    1157067          0       9655
  5   33876189     636041          0          0         60         60       4396       4396          0     627129          0      27488
  6   10663472    1364583          0          0          5          5       1426       1426          0    1361721          0      20341
  7   38937063     921272          0          0         54         54       5023       5023          0     911118          0      34159
  8   16044621       3966          0          0          9          9       1974       1974          0          0          0      46740
  9    2236927        610          0          0          0          0        305        305          0          0          0       3583
 10   23087850       5792          0          0          6          6       2890       2890          0          0          0      28946
 11   13996775       3362          0          0          8          8       1673       1673          0          0          0       8557
 12    9568717       2316          0          0          4          4       1154       1154          0          0          0      40620
 13    2488330        626          0          0          0          0        313        313          0          0          0       2902
 14    9604847       2392          0          0          6          6       1190       1190          0          0          0      27522
 15   18842212       4714          0          0          9          9       2348       2348          0          0          0       6366
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.094s (  938.588 blocks/s,  58.662 Mb/s)
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Stats6.png
Stats6.png
Attachment 11: Bias6.png
Bias6.png
  331   Wed May 19 11:24:55 2021 RDPMonday 19 April 12 noon shift
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2


		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd056 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x16 : 	 22  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      2      0      5      5      1      2      3      3      6   : 36120
aida02 :     25     10      0      4      3      5      2      3      3      3      6   : 36980
aida03 :     22     12      5      2      1      3      2      3      3      3      6   : 36616
aida04 :     16      3      3      3      4      4      4      2      3      3      6   : 36840
aida05 :     14      5      6      2      3      4      4      2      3      3      6   : 36800
aida06 :     30     10      6      3      3      4      2      3      3      3      6   : 36936
aida07 :     27     11      1      0      0      3      2      3      3      3      6   : 36436
aida08 :     38      7      7      3      3      2      2      3      3      3      6   : 36704
aida09 :      4      6      4      2      4      4      1      4      3      3      6   : 37056
aida10 :     21     10      3      3      1      5      1      3      3      3      6   : 36596
aida11 :     24      8      2      1      4      3      2      4      2      3      6   : 36192
aida12 :     31     12      4      3      5      5      2      2      3      3      6   : 36668

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Mon Apr 19 10:32:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

There should be a separate elog note about changing one of the bias voltages.

All histograms zeroed at 16:13.

16.20 DAQ continues file NULL/R33_156
      aida09 HEC fast comparator 0x2->0x1
      DSSSD#3 bias -100->-120V
Attachment 1: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 2: Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Attachment 3: 62.png
62.png
  330   Wed May 19 09:12:52 2021 Muneerah, Betool, LS17th May 00:00-12:00
Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
      Statistics ok - 210517_0202_Stats
      Temperatures ok - 210517_0202_Temp
      Bias and leakage ok - 210517_0200_Bias 
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155d : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      8      3      3      3      5      2      2      2      3      7   : 39460
aida02 :     22      4      5      0      3      2      2      3      2      3      7   : 39560
aida03 :     21     10      2      1      1      3      2      4      2      3      7   : 40100
aida04 :     18      7      2      0      3      3      3      2      2      3      7   : 39392
aida05 :     16      6      5      1      2      4      3      2      2      3      7   : 39520
aida06 :     18      6      6      2      2      2      2      3      2      3      7   : 39576
aida07 :     12      8      6      1      2      3      1      3      2      3      7   : 39408
aida08 :     13     11      4      0      1      3      3      2      2      3      7   : 39308
aida09 :     10      8      4      2      3      4      2      3      2      3      7   : 39848
aida10 :     16      8      2      1      2      4      2      3      1      4      7   : 40768
aida11 :     26     10      2      2      1      2      2      3      2      3      7   : 39512
aida12 :     16      9      2      1      2      4      2      2      2      3      7   : 39240
aida13 :     15      9      3      2      2      4      2      2      2      3      7   : 39284
aida14 :     25     10      2      3      3      3      2      3      2      3      7   : 39796
aida15 :      4      2      2      1      1      3      2      2      2      3      7   : 38944
aida16 :     16      7      3      3      1      3      3      2      2      3      7   : 39368

04:00   Statistics ok - 210517_0403_Stats
        Temperatures ok - 210517_0403_Temp
        Bias and leakage ok - 210517_0400_Bias 
04:05 System wide checks:
      Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1560 : 	 14  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      7      5      2      3      5      2      2      2      3      7   : 39492
aida02 :     23      6      7      2      1      3      2      3      2      3      7   : 39676
aida03 :     17      9      3      2      0      3      2      3      2      3      7   : 39548
aida04 :     16      4      1      2      1      3      3      3      2      3      7   : 39792
aida05 :     22      7      3      3      1      4      3      3      2      3      7   : 40032
aida06 :     18      6      7      3      1      3      2      3      2      3      7   : 39688
aida07 :     18      7      4      1      2      4      1      4      2      3      7   : 40032
aida08 :     19      5      4      1      1      3      3      3      2      3      7   : 39828
aida09 :     10      6      7      1      3      4      2      2      2      3      7   : 39336
aida10 :     12     10      2      3      1      4      2      3      1      4      7   : 40768
aida11 :     10      9      4      3      2      1      2      3      2      3      7   : 39440
aida12 :      8      7      6      1      2      4      2      3      2      3      7   : 39768
aida13 :     11      2      6      1      2      3      2      3      2      3      7   : 39612
aida14 :     27      7      4      2      3      3      2      3      2      3      7   : 39780
aida15 :      4      2      3      1      1      3      2      2      2      3      7   : 38960
aida16 :     18      8      4      2      1      4      2      2      2      3      7   : 39240


06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
 	Statistics ok - 210517_0742_Stats
        Temperatures ok - 210517_0743_Temp
        Bias and leakage ok - 210517_0740_Bias 

System wide checks

	
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0x61da : 	 0x61dd : 	 3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      6      3      1      1      3      1      3      3      3      7   : 40336
aida02 :     11      6      0      1      1      5      2      4      2      3      7   : 40252
aida03 :      0      5      2      3      3      3      3      4      2      3      7   : 40424
aida04 :      2      4      2      1      2      4      3      3      2      3      7   : 39912
aida05 :     11      7      5      1      2      2      4      2      3      3      7   : 40532
aida06 :     20      6      0      2      3      2      1      3      3      3      7   : 40320
aida07 :     15      5      1      2      2      3      2      2      3      3      7   : 40116
aida08 :      2      3      4      1      3      3      2      2      3      3      7   : 40128
aida09 :     18      5      1      3      2      4      2      2      3      3      7   : 40288
aida10 :     11      6      3      4      2      3      3      2      2      4      7   : 41484
aida11 :      4      4      5      2      2      2      3      3      2      3      7   : 39744
aida12 :     18      7      2      3      2      2      4      3      2      3      7   : 40064
aida13 :      2      2      1      3      0      3      2      4      2      3      7   : 39944
aida14 :     16      4      5      3      3      3      3      2      2      4      7   : 41552
aida15 :      9      3      1      1      2      3      2      3      3      3      7   : 40556
aida16 :     16      5      1      2      1      2      3      4      2      3      7   : 40184

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs


12:52 Check FPGA Timestamp errors gives following error message:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds

Dead time check:

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252995858 (  451517.9 Hz)
        Other data format:    8924143 (   15926.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.2 Hz)
                               RESUME:         85 (       0.2 Hz)
                              SYNC100:      34087 (      60.8 Hz)
                              WR48-63:      34087 (      60.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8855799 (   15804.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      560.323 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000              349.960
  1                0.002                0.000
  2                0.000                5.974
  3                1.105                0.000
  4                0.000              447.454
  5                0.000                2.809
  6                0.000               16.727
  7                0.137                0.000
  8                0.090                0.000
  9                0.000              429.897
 10                0.461                0.000
 11                0.018                0.000
 12                0.000                8.985
 13                0.000              466.311
 14                0.000               15.095
 15                0.346                0.000

13:24 Beam is off at the moment...

13:43 Changing spill structure from 1.5s/3.5s to 1s. 

13:50

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251763810 (  930342.1 Hz)
        Other data format:   10156190 (   37530.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        205 (       0.8 Hz)
                               RESUME:        205 (       0.8 Hz)
                              SYNC100:      33009 (     122.0 Hz)
                              WR48-63:      33009 (     122.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10089762 (   37284.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527077 (    1947.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      270.614 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.034                6.803
  1                8.461                0.000
  2                0.569                0.000
  3               11.664                0.000
  4                0.024               26.416
  5                0.854                0.000
  6                0.069                0.000
  7                0.741                0.000
  8                1.336                0.000
  9                0.000               75.926
 10                0.353                0.000
 11                0.105                0.000
 12                0.070                0.000
 13                0.000               93.585
 14                0.017                0.000
 15                0.121                0.000

15:06 Spill structure back to 1.5 on 2s off.
Attachment 1: 210517_0200_Bias.png
210517_0200_Bias.png
Attachment 2: 210517_0202_stats.png
210517_0202_stats.png
Attachment 3: 210517_0202_temp.png
210517_0202_temp.png
Attachment 4: 210517_0400_Bias.png
210517_0400_Bias.png
Attachment 5: 210517_0403_stats.png
210517_0403_stats.png
Attachment 6: 210517_0403_temp.png
210517_0403_temp.png
Attachment 7: 210517_0740_Bias.png
210517_0740_Bias.png
Attachment 8: 210517_0742_Stats.png
210517_0742_Stats.png
Attachment 9: 210517_0743_Temp.png
210517_0743_Temp.png
Attachment 10: 0833_Stats.png
0833_Stats.png
Attachment 11: 0902_Stats.png
0902_Stats.png
Attachment 12: 1241_Stats.png
1241_Stats.png
Attachment 13: 1347_Stats.png
1347_Stats.png
Attachment 14: FEETemps_1042.png
FEETemps_1042.png
Attachment 15: FEETemps_1246.png
FEETemps_1246.png
Attachment 16: FEETemps_1350.png
FEETemps_1350.png
Attachment 17: leakage_currents_1302.png
leakage_currents_1302.png
  329   Wed May 19 09:09:27 2021 MS, JS, OHMay 16 08:00-24:00
16th May 08:00 - 12:00 shift
Author: MS


08:00


	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7163 : 	 23  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      7      2      3      2      3      2      4      2      3      7   : 40228
aida02 :      3      8      3      2      2      2      2      4      2      3      7   : 39996
aida03 :     23      9      6      1      0      3      2      4      2      3      7   : 40100
aida04 :     34     27     17      7      2      4      3      3      2      2      7   : 38608
aida05 :     19      9      5      2      2      2      3      3      2      3      7   : 39844
aida06 :      5      5      2      1      0      4      2      4      2      3      7   : 40060
aida07 :     28      2     10      3      2      4      1      4      2      3      7   : 40192
aida08 :     19      5      5      1      2      5      1      3      2      3      7   : 39652
aida09 :     20      8      3      3      1      3      2      3      2      3      7   : 39648
aida10 :     27     10      0      2      2      2      2      3      1      4      7   : 40572
aida11 :      3      3      2      1      2      4      2      3      2      3      7   : 39652
aida12 :     16      7     11      2      3      4      2      2      2      3      7   : 39464
aida13 :     14     10      4      3      1      5      2      2      2      3      7   : 39400
aida14 :     21      9     10      1      1      2      2      3      1      4      7   : 40604
aida15 :     19      8      2      3      0      4      3      2      2      3      7   : 39436
aida16 :     24      5      8      3      3      4      2      2      2      3      7   : 39464







 *** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000






10:00


	
FEE64 module aida06 global clocks failed, 6
 FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7166 : 	 26  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      7      6      2      2      3      2      3      2      3      7   : 39716
aida02 :      8      4      3      2      2      3      2      3      2      3      7   : 39600
aida03 :     23     11      5      1      0      4      2      3      1      4      7   : 40740
aida04 :     42     25     16      7      2      4      4      3      2      2      7   : 38864
aida05 :     24      5      5      1      2      2      3      3      1      4      7   : 40824
aida06 :     11      4      4      1      1      4      2      4      2      3      7   : 40172
aida07 :     21      8      5      1      2      3      2      4      2      3      7   : 40196
aida08 :     15      5      6      1      1      4      2      4      2      3      7   : 40228
aida09 :     15     10      4      1      2      3      2      3      2      3      7   : 39660
aida10 :     23      8      2      2      2      2      2      3      1      4      7   : 40572
aida11 :      6      4      4      2      2      4      2      3      2      3      7   : 39736
aida12 :     18      8      8      1      4      4      3      2      2      3      7   : 39720
aida13 :     23      6      2      3      2      5      2      2      2      3      7   : 39436
aida14 :     29      3     11      1      1      2      2      3      1      4      7   : 40604
aida15 :     29      7      2      2      0      4      3      3      2      3      7   : 39948
aida16 :      2      3      6      2      3      4      2      2      2      3      7   : 39296






*** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS

11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok  06.410 uA      006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%

PAUSE: 166  RESUME: 166

 *** Timestamp elapsed time:      196.305 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.044                0.000
  1               12.405                0.000
  2                0.807                0.000
  3                7.260                0.000
  4                0.014                0.000
  5                0.458                0.000
  6                0.027                0.000
  7                0.497                0.000
  8                0.723                0.000
  9                0.000               88.857
 10                6.189                0.000
 11                1.443                0.000
 12                0.474                0.000
 13                0.000               35.565
 14                0.000                0.000
 15                0.147                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x716e : 	 34  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      7      4      2      2      3      2      3      1      4      7   : 40712
aida02 :     25     10      7      1      1      4      1      3      2      3      7   : 39556
aida03 :     22     10      5      1      0      4      2      4      2      3      7   : 40216
aida04 :     40     24     17      7      2      5      3      3      2      2      7   : 38736
aida05 :      4      8      3      0      1      3      3      3      1      4      7   : 40768
aida06 :     21      6      6      2      2      3      2      4      2      3      7   : 40228
aida07 :     25      4      6      3      2      3      2      4      2      3      7   : 40260
aida08 :     19      9      4      1      1      4      2      4      2      3      7   : 40244
aida09 :     16      9      4      2      2      3      2      3      2      3      7   : 39688
aida10 :     19     10      5      1      2      2      2      4      1      4      7   : 41100
aida11 :     21     10      2      1      2      3      3      3      2      3      7   : 39908
aida12 :     25      7      7      2      2      3      2      3      2      3      7   : 39756
aida13 :     13      7      3      4      2      5      2      3      2      3      7   : 39964
aida14 :     21      7      9      2      1      2      2      4      1      4      7   : 41116
aida15 :     23      6      2      3      0      4      3      3      2      3      7   : 39948
aida16 :      9     11     10      2      3      4      2      2      2      3      7   : 39452

Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.

12:35 -
usbec ok.
Current ok 
Stats good 
Temps ok 
Analysis ok R7_395. Dead time FEE1 still high 6.6%

13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok    006.850 uA      007.250 uA 
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok 
Analysis R7_415. Dead time FEE1 & FEE10 high 10%

14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast -  Aida11 low - 

14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.

15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.

18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
      During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
      The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
      We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
      During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.

18:08 System wide checks all ok - bar some ADC but waveforms disabled
      Statistics ok - 210516_1809_Stats
      Temperatures ok - 210516_1809_Temp
      Bias and leakage ok - 210516_1810_Bias

18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.

19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
      Now have around 4.2TB left which will provide around 80 hours of writing

20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.

20:57 System wide checks. Clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1556 : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics ok - 210516_2056_Stats
      Temp ok -  210516_2058_Temp
      Bias and leakage current ok -  210516_2058_Bias

23:16 System wide checks:
      Clock still ok 
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155a : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

       Statistics - 210516_2315_Stats
       Temperature - 210516_2316_Temp
       Bias and leakage current ok - 210516_231
Attachment 1: Screenshot_2021-05-16_Statistics_aidas-gsi(4).png
Screenshot_2021-05-16_Statistics_aidas-gsi(4).png
Attachment 2: Screenshot_2021-05-16_Statistics_aidas-gsi(5).png
Screenshot_2021-05-16_Statistics_aidas-gsi(5).png
Attachment 3: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 4: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png
Attachment 5: 1Statistics_aidas-gsi(6).png
1Statistics_aidas-gsi(6).png
Attachment 6: 1Temperature_and_status_scan_aidas-gsi(6).png
1Temperature_and_status_scan_aidas-gsi(6).png
Attachment 7: Bias1.png
Bias1.png
Attachment 8: 210516_0857_Temp.png
210516_0857_Temp.png
Attachment 9: 210516_1554_Layout8.png
210516_1554_Layout8.png
Attachment 10: 210516_1555_Layout7.png
210516_1555_Layout7.png
Attachment 11: 210516_1600_Layout1.png
210516_1600_Layout1.png
Attachment 12: 210516_1601_Stats.png
210516_1601_Stats.png
Attachment 13: 210516_1605_1606.png
210516_1605_1606.png
Attachment 14: 210516_1808_Stats.png
210516_1808_Stats.png
Attachment 15: 210516_1809_Temp.png
210516_1809_Temp.png
Attachment 16: 210516_1810_Bias.png
210516_1810_Bias.png
Attachment 17: 210516_2056_Stats.png
210516_2056_Stats.png
Attachment 18: 210516_2058_Bias.png
210516_2058_Bias.png
Attachment 19: 210516_2313_Stats.png
210516_2313_Stats.png
Attachment 20: 210516_2315_Temp.png
210516_2315_Temp.png
Attachment 21: 210516_2316_Bias.png
210516_2316_Bias.png
  328   Wed May 19 09:06:58 2021 CAMay 16th 00:00 - 08:00
16th May 2021 - 00:00 - 08:00 Shift
Author: CA

00:15 System wide checks;

FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7154 : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19     10      3      1      2      3      2      3      2      3      7   : 39660
aida02 :     29      5      3      1      1      3      2      3      2      3      7   : 39596
aida03 :     15      7      1      1      2      3      2      3      2      3      7   : 39588
aida04 :     44     26     19      5      2      4      3      3      2      2      7   : 38608
aida05 :     20     10      3      0      1      4      2      2      3      3      7   : 40208
aida06 :     27      6      2      1      2      4      2      4      2      3      7   : 40284
aida07 :     25      5      3      1      2      3      2      3      2      3      7   : 39644
aida08 :     23      8      2      3      1      4      2      3      2      3      7   : 39772
aida09 :     23     10      3      2      3      4      1      3      2      3      7   : 39644
aida10 :     18     10      3      2      3      2      2      2      2      4      7   : 41160
aida11 :     16      7      3      2      1      2      2      3      2      3      7   : 39464
aida12 :     21      4      7      1      3      3      3      3      2      3      7   : 40004
aida13 :     25      6      3      2      1      3      1      4      2      3      7   : 39876
aida14 :     20      8     10      1      1      2      2      2      2      4      7   : 41104
aida15 :     26      6      3      3      0      4      3      2      2      3      7   : 39464
aida16 :      9      8      2      4      1      3      1      4      2      3      7   : 39876

00:30 Stats ok - 210516_0030_stats.png
      Temps ok - 210516_0030_temps.png
      Bias ok - 210516_0030_bias.png

00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt

02:21 system wide checks as above, except;

      		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7159 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x7 : 	 7  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:28 Stats ok - 210516_0230_stats.png
      Temps ok - 210516_0230_temps.png
      Bias ok - 210516_0230_bias.png

02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt

04:36 system wide checks same as above, except;

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x715c : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x9 : 	 9  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:37 Stats ok - 210516_0430_stats.png
      Temps ok - 210516_0430_temps.png
      Bias ok - 210516_0430_bias.png

04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt

06:38 system wide checks same as before, except;

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7161 : 	 21  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:40 Stats ok - 210516_0640_stats.png
      Temps ok - 210516_0640_temps.png
      Bias ok - 210516_0640_bias.png

06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt
Attachment 1: 210516_0030_bias.png
210516_0030_bias.png
Attachment 2: 210516_0030_stats.png
210516_0030_stats.png
Attachment 3: 210516_0030_temps.png
210516_0030_temps.png
Attachment 4: 210516_0230_bias.png
210516_0230_bias.png
Attachment 5: 210516_0230_stats.png
210516_0230_stats.png
Attachment 6: 210516_0230_temps.png
210516_0230_temps.png
Attachment 7: 210516_0430_bias.png
210516_0430_bias.png
Attachment 8: 210516_0430_stats.png
210516_0430_stats.png
Attachment 9: 210516_0430_temps.png
210516_0430_temps.png
Attachment 10: 210516_0640_bias.png
210516_0640_bias.png
Attachment 11: 210516_0640_stats.png
210516_0640_stats.png
Attachment 12: 210516_0640_temps.png.png
210516_0640_temps.png.png
Attachment 13: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255660963 ( 1413413.6 Hz)
        Other data format:    6259037 (   34602.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        192 (       1.1 Hz)
                               RESUME:        192 (       1.1 Hz)
                              SYNC100:      32674 (     180.6 Hz)
                              WR48-63:      32674 (     180.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6193305 (   34239.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     287054 (    1587.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      180.882 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.117                0.000
  1                8.836                0.000
  2                1.027                0.000
  3                5.545                0.000
  4                0.000                5.604
  5                0.873                0.000
  6                0.370                0.000
  7                0.861                0.000
  8                0.767                0.000
  9                0.000               63.434
 10               13.201                0.000
 11                3.930                0.000
 12                0.813                0.000
 13                0.000               30.131
 14                0.448                0.000
 15                1.895                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9469697       3614          0          0          3          3       1264       1264          0       1080          0      15256
  1   23268933       6078          0          0         41         41       2998       2998          0          0          0      31257
  2   14135373    1756135          0          0          5          5       1968       1968          0    1752189          0      25654
  3   26646926    1718468          0          0         28         28       3507       3507          0    1711398          0      36818
  4    2828773     749772          0          0          0          0        426        426          0     748920          0       7104
  5   14183721     453202          0          0          5          5       1830       1830          0     449532          0      20813
  6   11921192     886700          0          0          5          5       1583       1583          0     883524          0      13410
  7   19791515     651950          0          0          5          5       2639       2639          0     646662          0      25026
  8   22244739       5464          0          0          4          4       2728       2728          0          0          0      32840
  9    1601175        378          0          0          0          0        189        189          0          0          0       2522
 10   41869813      10332          0          0         63         63       5103       5103          0          0          0      18083
 11   18332381       4602          0          0         16         16       2285       2285          0          0          0       5732
 12   14937821       3786          0          0          7          7       1886       1886          0          0          0      28026
 13    2220427        536          0          0          0          0        268        268          0          0          0       2121
 14   11574320       2908          0          0          3          3       1451       1451          0          0          0      17899
 15   20634157       5112          0          0          7          7       2549       2549          0          0          0       4493
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   47.672s (  671.255 blocks/s,  41.953 Mb/s)
Attachment 14: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255713262 ( 1398395.0 Hz)
        Other data format:    6206738 (   33942.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        185 (       1.0 Hz)
                               RESUME:        185 (       1.0 Hz)
                              SYNC100:      32681 (     178.7 Hz)
                              WR48-63:      32681 (     178.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6141006 (   33582.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     279005 (    1525.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      182.862 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.488                0.000
  1                8.465                0.000
  2                0.216                0.000
  3                7.120                0.000
  4                0.011                8.896
  5                0.764                0.000
  6                0.401                0.000
  7                1.395                0.000
  8                0.721                0.000
  9                0.000               86.292
 10               11.881                0.000
 11                1.912                0.000
 12                0.187                0.000
 13                0.000               15.181
 14                0.528                0.000
 15                0.095                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9424415       3624          0          0          2          2       1264       1264          0       1092          0      14964
  1   23109905       5922          0          0         39         39       2922       2922          0          0          0      29876
  2   14322958    1782809          0          0          3          3       2029       2029          0    1778745          0      25108
  3   26234410    1716299          0          0         35         35       3458       3458          0    1709313          0      35100
  4    2817740     734656          0          0          1          1        433        433          0     733788          0       6814
  5   13978921     443934          0          0          4          4       1871       1871          0     440184          0      20242
  6   11924553     862150          0          0          4          4       1621       1621          0     858900          0      13000
  7   19324336     623898          0          0         11         11       2446       2446          0     618984          0      24043
  8   21963572       5366          0          0          5          5       2678       2678          0          0          0      32304
  9    1580877        372          0          0          0          0        186        186          0          0          0       2567
 10   42874488      10618          0          0         59         59       5250       5250          0          0          0      17549
 11   18504188       4566          0          0         14         14       2269       2269          0          0          0       5838
 12   14580871       3684          0          0          2          2       1840       1840          0          0          0      27687
 13    2240991        612          0          0          0          0        306        306          0          0          0       2138
 14   11530867       2858          0          0          4          4       1425       1425          0          0          0      17152
 15   21300170       5370          0          0          2          2       2683       2683          0          0          0       4623
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   45.405s (  704.764 blocks/s,  44.048 Mb/s)
Attachment 15: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255591110 ( 1400708.0 Hz)
        Other data format:    6328890 (   34684.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        179 (       1.0 Hz)
                               RESUME:        179 (       1.0 Hz)
                              SYNC100:      32680 (     179.1 Hz)
                              WR48-63:      32680 (     179.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6263172 (   34323.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     288458 (    1580.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      182.473 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                7.649                0.000
  2                0.265                0.000
  3                5.929                0.000
  4                0.000               21.293
  5                1.614                0.000
  6                0.390                0.000
  7                0.004                0.000
  8                1.749                0.000
  9                0.000               75.748
 10               12.608                0.000
 11                1.662                0.000
 12                0.334                0.000
 13                0.000               39.700
 14                0.092                0.000
 15                1.009                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9481493       3580          0          0          0          0       1244       1244          0       1092          0      15504
  1   23234003       5924          0          0         42         42       2920       2920          0          0          0      30950
  2   14360490    1787895          0          0          2          2       2026       2026          0    1783839          0      26238
  3   26580599    1734745          0          0         32         32       3537       3537          0    1727607          0      36830
  4    2867864     759325          0          0          0          0        449        449          0     758427          0       7075
  5   13998560     454755          0          0          6          6       1806       1806          0     451131          0      20732
  6   12019427     893028          0          0          3          3       1644       1644          0     889734          0      13550
  7   19745451     656494          0          0          1          1       2575       2575          0     651342          0      25286
  8   21649633       5478          0          0          7          7       2732       2732          0          0          0      32719
  9    1602793        374          0          0          0          0        187        187          0          0          0       2586
 10   42457575      10646          0          0         62         62       5261       5261          0          0          0      18276
 11   18423606       4634          0          0         16         16       2301       2301          0          0          0       6050
 12   14493303       3540          0          0          2          2       1768       1768          0          0          0      27878
 13    2240564        512          0          0          0          0        256        256          0          0          0       2153
 14   11446056       2794          0          0          2          2       1395       1395          0          0          0      17997
 15   20989693       5166          0          0          4          4       2579       2579          0          0          0       4634
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   52.041s (  614.900 blocks/s,  38.431 Mb/s)
Attachment 16: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254484788 ( 1267953.1 Hz)
        Other data format:    7435212 (   37045.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        166 (       0.8 Hz)
                               RESUME:        166 (       0.8 Hz)
                              SYNC100:      32747 (     163.2 Hz)
                              WR48-63:      32747 (     163.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7369386 (   36717.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     359272 (    1790.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      200.705 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.411                0.000
  1               11.545                0.000
  2                1.046                0.000
  3                6.943                0.000
  4                0.008                8.993
  5                0.438                0.000
  6                0.000                0.000
  7                1.702                0.000
  8                0.655                0.000
  9                0.000               75.734
 10                6.237                0.000
 11                1.119                0.000
 12                0.217                0.000
 13                0.000               28.278
 14                0.250                0.000
 15                0.105                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8401665       3348          0          0          2          2       1072       1072          0       1200          0      19564
  1   26165949       6958          0          0         57         57       3422       3422          0          0          0      37826
  2   13624578    2004081          0          0          4          4       1973       1973          0    2000127          0      32349
  3   29939387    1947767          0          0         35         35       4004       4004          0    1939689          0      45332
  4    3232848     946545          0          0          1          1        548        548          0     945447          0       8745
  5   15270389     573894          0          0          6          6       1986       1986          0     569910          0      26107
  6   10372112    1114231          0          0          0          0       1457       1457          0    1111317          0      17063
  7   19130864     806574          0          0          8          8       2431       2431          0     801696          0      31336
  8   24018310       5808          0          0          5          5       2899       2899          0          0          0      41117
  9    1787485        418          0          0          0          0        209        209          0          0          0       3357
 10   42125761      10654          0          0         29         29       5298       5298          0          0          0      22828
 11   16187533       3960          0          0         11         11       1969       1969          0          0          0       7397
 12   14311704       3490          0          0          1          1       1744       1744          0          0          0      35224
 13    2388212        620          0          0          0          0        310        310          0          0          0       2664
 14    9736084       2414          0          0          1          1       1206       1206          0          0          0      22553
 15   17791907       4450          0          0          6          6       2219       2219          0          0          0       5810
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.287s (  739.250 blocks/s,  46.203 Mb/s)
  327   Wed May 19 06:59:33 2021 OHWednesday 19th May 08:00 - 16:00
08:00 OH Takes over for CA
      Thresholds 0xc for p+n and 0x19 for n+n ASICs 1-3 and 0x64 for n+nASIC4

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

       Max average dead time in R14_586 3.2% which is 9.6% on spill
       Stable from yesterday

10:02 System wide checks. All ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x554 : 	 84  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics - Attachment 1
      Temp - attachment 2
      Bias and leakage currents ok - Attachment 3


11:17 A new run was started as the microspill has been changed. I believe to optimise FRS deadtime.
11:45 They are still working on it

12:05 Beam off for a bit while they optimise spill

12:15 System wide checks ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - Attachment 4
     Temp - Attachment 5
     Bias - Attachment 6

13:11 Magda has said that the changes to the spill structure only gained them a few percent in dead time
      Further changes could be done, but would take at least two hours

13:19 Dead time following the current spill changes
	R14_648	R14_649	R14_650	R14_651	Sum DT (S)   Avergage DT (%)   On Spill DT (%)
FEE	244.823	252.168	247.155	240.421	984.567		
0	0.063	0.025	0.105	0.096	0.289	     0.029353005	0.088059015
1	3.981	4.907	4.17	2.484	15.542	     1.578561947	4.73568584
2	0.01	0	0.121	0.779	0.91	     0.092426417	0.277279251
3	3.625	2.972	2.766	3.335	12.698	     1.289704002	3.869112006
4	0.088	0.1	0.12	0.136	0.444	     0.045095966	0.135287898
5	9.392	5.798	9.835	8.01	33.035	     3.355282068	10.0658462
6	0.153	0.075	0.01	0.517	0.755	     0.076683456	0.230050367
7	7.4	7.565	7.703	6.415	29.083	     2.953887343	8.86166203
8	0.221	0.606	1.321	0.539	2.687	     0.272911849	0.818735546
9	0	0	0	0	0	     0          	0
10	1.381	0.368	1.196	1.091	4.036	     0.409926394	1.229779182
11	0.066	0.025	0.039	0.017	0.147	     0.014930421	0.044791264
12	0.166	0.041	0.062	0.024	0.293	     0.029759275	0.089277825
13	0	0	0	0	0	     0          	0
14	0.71	0	0.005	0.074	0.789	     0.08013675	        0.240410251
15	0.239	0.17	0.208	0.286	0.903	     0.091715444	0.275146333
16	0	0	0	0	0	     0          	0

14:03 System wide checks ok - WR same difference as before
      Statistics - attachment 7
      Temperatures - attachment 8
      Bias and leakage currents - attachment 9
Attachment 1: 210519_1002_Stats.png
210519_1002_Stats.png
Attachment 2: 210519_1002_Temp.png
210519_1002_Temp.png
Attachment 3: 210519_1003_Bias.png
210519_1003_Bias.png
Attachment 4: 210519_1213_Stats.png
210519_1213_Stats.png
Attachment 5: 210519_1214_Temp.png
210519_1214_Temp.png
Attachment 6: 210519_1215_Bioas.png
210519_1215_Bioas.png
Attachment 7: 210519_1403_Stats.png
210519_1403_Stats.png
Attachment 8: 210519_1404_Temp.png
210519_1404_Temp.png
Attachment 9: 210519_1405_bias.png
210519_1405_bias.png
  326   Tue May 18 23:11:40 2021 CAWednesday May 19th 00:00 - 08:00
00:00 CA takes over
      LS has performed usual checks, see previous Elog entry - all ok

00:30 stats ok
      R14_470 deadtime at 4.4%

02:10 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x52f : 	 47  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

02:12 FEE64 Temps ok - attachment 1
      Stats ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      dead-times ok - R14_497 max 4% - attachment 4

03:27 DAQ continues ok - writing to file R14_517

04:07 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x537 : 	 55  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temps ok - attachment 6
      Stats ok - attachment 5
      detector bias / leakage currents ok - attachment 7
      dead-times ok - R14_526 max 4% - attachment 8

05:27 bad timestamp error in merger terminal - attachment 9

06:05 all system wide checks ok except;

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x53d : 	 61  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:07 FEE64 Temps ok - attachment 10
      Stats ok - attachment 11
      detector bias / leakage currents ok - attachment 12
      dead-times ok - R14_558 max 3.6% - attachment 13

07:01 another bad timestamp error in merger terminal - attachment 14
Attachment 1: Screenshot_from_2021-05-19_01-13-48.png
Screenshot_from_2021-05-19_01-13-48.png
Attachment 2: Screenshot_from_2021-05-19_01-14-09.png
Screenshot_from_2021-05-19_01-14-09.png
Attachment 3: Screenshot_from_2021-05-19_01-14-24.png
Screenshot_from_2021-05-19_01-14-24.png
Attachment 4: Screenshot_from_2021-05-19_01-19-37.png
Screenshot_from_2021-05-19_01-19-37.png
Attachment 5: Screenshot_from_2021-05-19_03-04-39.png
Screenshot_from_2021-05-19_03-04-39.png
Attachment 6: Screenshot_from_2021-05-19_03-05-33.png
Screenshot_from_2021-05-19_03-05-33.png
Attachment 7: Screenshot_from_2021-05-19_03-05-53.png
Screenshot_from_2021-05-19_03-05-53.png
Attachment 8: Screenshot_from_2021-05-19_03-09-03.png
Screenshot_from_2021-05-19_03-09-03.png
Attachment 9: Screenshot_from_2021-05-19_04-26-09.png
Screenshot_from_2021-05-19_04-26-09.png
Attachment 10: Screenshot_from_2021-05-19_05-08-01.png
Screenshot_from_2021-05-19_05-08-01.png
Attachment 11: Screenshot_from_2021-05-19_05-08-23.png
Screenshot_from_2021-05-19_05-08-23.png
Attachment 12: Screenshot_from_2021-05-19_05-08-46.png
Screenshot_from_2021-05-19_05-08-46.png
Attachment 13: Screenshot_from_2021-05-19_05-11-33.png
Screenshot_from_2021-05-19_05-11-33.png
Attachment 14: Screenshot_from_2021-05-19_06-12-29.png
Screenshot_from_2021-05-19_06-12-29.png
  325   Tue May 18 18:47:25 2021 LSTuesday 18th May 20:00 - 24:00
19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x520 : 	 32  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~1.8M items/s
Tape server ~7MB/s

Analysed R14_399, deadtimes are okay (Fig 4)

20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)

21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)

21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)

22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x524 : 	 36  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.3M items/s
Tape server ~8MB/s

Analysed R14_430, deadtimes are okay (Fig 14)

22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%

23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)

23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)

24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x529 : 	 41  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

The difference has increased over time (not sure if an issue)

Merger rate ~3M items/s
Tape server ~8MB/s

Analysed R14_462, deadtimes are okay (Fig 24)
Attachment 1: 210518_Stats_1950.png
210518_Stats_1950.png
Attachment 2: 210518_FEEtemps_1950.png
210518_FEEtemps_1950.png
Attachment 3: 210518_Bias_1950.png
210518_Bias_1950.png
Attachment 4: 210518_Deadtimes_R14_399.png
210518_Deadtimes_R14_399.png
Attachment 5: 210518_Stats_2028.png
210518_Stats_2028.png
Attachment 6: 210518_Deadtimes_R14_407.png
210518_Deadtimes_R14_407.png
Attachment 7: 210518_Stats_2059.png
210518_Stats_2059.png
Attachment 8: 210518_Deadtimes_R14_415.png
210518_Deadtimes_R14_415.png
Attachment 9: 210518_Stats_2130.png
210518_Stats_2130.png
Attachment 10: 210518_Deadtimes_R14_422.png
210518_Deadtimes_R14_422.png
Attachment 11: 210518_Stats_2158.png
210518_Stats_2158.png
Attachment 12: 210518_FEEtemps_2158.png
210518_FEEtemps_2158.png
Attachment 13: 210518_bias_2158.png
210518_bias_2158.png
Attachment 14: 210518_Deadtimes_R14_430.png
210518_Deadtimes_R14_430.png
Attachment 15: 210518_Stats_2230.png
210518_Stats_2230.png
Attachment 16: 210518_deadtimes_R14_438.png
210518_deadtimes_R14_438.png
Attachment 17: 210518_Stats_2300.png
210518_Stats_2300.png
Attachment 18: 210518_Deadtimes_R14_445.png
210518_Deadtimes_R14_445.png
Attachment 19: 210518_Stats_2330.png
210518_Stats_2330.png
Attachment 20: 210518_Deadtimes_R14_454.png
210518_Deadtimes_R14_454.png
Attachment 21: 210518_Stats_2400.png
210518_Stats_2400.png
Attachment 22: 210518_FEEtemps_2400.png
210518_FEEtemps_2400.png
Attachment 23: 210518_bias_2400.png
210518_bias_2400.png
Attachment 24: 210518_Deadtimes_R14_462.png
210518_Deadtimes_R14_462.png
  324   Tue May 18 14:57:43 2021 CBTuesday 18th May 16:00 - 20:00
15:58 Took shift over from OH and PP

Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x510 : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     22      8      7      4      1      4      2      2      2      2      7   : 37320
aida02 :     12      9      4      2      4      2      3      3      3      3      6   : 36856
aida03 :     19      6      7      2      2      2      2      4      3      3      6   : 37036
aida04 :     18      9     10      6      1      4      1      4      3      3      6   : 37168
aida05 :     17     12      2      1      2      3      3      3      3      3      6   : 36836
aida06 :     19      8      7      7      2      4      2      4      3      3      6   : 37468
aida07 :     28      5      4      1      2      4      2      3      2      2      7   : 37752
aida08 :     17      5      4      2      1      2      2      3      3      3      6   : 36396
aida09 :     21     10      4      5      4      4      3      3      3      3      6   : 37252
aida10 :     15     13      4      2      2      2      3      2      2      4      6   : 37284
aida11 :     22     13      3      3      4      3      1      4      3      3      6   : 37072
aida12 :      2      8      1      3      1      3      2      3      3      3      6   : 36472
aida13 :     23      5      4      4      1      4      3      2      3      3      6   : 36484
aida14 :     19      7      4      3      1      3      2      4      2      4      6   : 38116
aida15 :     20     10      2      2      3      2      3      2      3      3      6   : 36288
aida16 :     29      6      1      2      0      3      1      4      3      3      6   : 36724

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

Analysed R14_342. Attach 4. Dead time <5%



17:00

Stats OK - attach 5
Temps OK - as before
Bias  increasing quite quickly - attach 6

System-wide checks

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x512 : 	 18  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     28      7      8      4      1      4      2      2      2      2      7   : 37352
aida02 :     24     11      6      4      3      2      3      2      3      3      6   : 36440
aida03 :     17     11      5      2      1      2      2      3      3      3      6   : 36460
aida04 :     25     13     10      6      2      4      2      3      3      3      6   : 37036
aida05 :     23      7      3      2      2      3      3      3      3      3      6   : 36868
aida06 :      1     12      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     22     12      1      1      2      4      2      2      2      2      7   : 37224
aida08 :     21      8      7      2      1      3      3      2      3      3      6   : 36356
aida09 :     23      7      4      6      4      3      2      4      3      3      6   : 37396
aida10 :     27      7      2      2      2      2      3      3      2      4      6   : 37764
aida11 :     12      5      3      5      4      3      1      4      3      3      6   : 37032
aida12 :     23      9      4      1      2      3      2      3      3      3      6   : 36612
aida13 :     10     12      6      3      1      4      3      2      3      3      6   : 36488
aida14 :     23      7      3      3      1      3      2      3      2      4      6   : 37604
aida15 :     22      6      6      4      2      2      3      2      3      3      6   : 36328
aida16 :     27      7      3      1      0      3      1      4      3      3      6   : 36724



18:00

Stats OK - attach 7
Temps OK - as before
Bias - attach 8

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x517 : 	 23  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_371. Attach 9. Dead time below 5%.



18:35 MBS DAQ crashed



18:40
Stats OK - attach 9
Temps OK - as before
Bias - attach 10

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51a : 	 26  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:42 MBS DAQ restarted - new file being opened.


19:36
Stats OK - attach 11
Temps OK - attach 12
Bias - attach 13

System-wide checks
Clock OK
ADC all fail

	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51e : 	 30  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA OK

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     24     11      5      4      1      4      2      3      2      2      7   : 37832
aida02 :      0      5      6      3      4      2      3      2      3      3      6   : 36328
aida03 :     15      8      5      3      2      2      1      3      4      3      6   : 37292
aida04 :     14      8      7      5      3      4      2      3      3      3      6   : 36936
aida05 :     31      6      1      1      2      3      3      3      2      4      6   : 37852
aida06 :      9      8      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     21      7      2      1      2      4      1      3      2      2      7   : 37452
aida08 :      8      4      5      3      1      3      2      3      3      3      6   : 36528
aida09 :     24      7      3      5      4      3      2      4      3      3      6   : 37352
aida10 :     19      9      7      1      2      2      3      2      2      4      6   : 37284
aida11 :     27      5      0      5      4      3      1      4      3      3      6   : 37044
aida12 :     31      9      2      1      2      3      2      3      3      3      6   : 36612
aida13 :     21     10      4      3      1      4      3      2      3      3      6   : 36484
aida14 :     22      6      4      2      1      3      2      4      2      4      6   : 38088
aida15 :     21      9      3      4      2      2      3      2      3      3      6   : 36300
aida16 :     23     11      4      2      1      4      2      3      3      3      6   : 36724


19:43 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_342
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252065796 ( 1015168.3 Hz)
        Other data format:    9854206 (   39686.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        242 (       1.0 Hz)
                               RESUME:        241 (       1.0 Hz)
                              SYNC100:      32924 (     132.6 Hz)
                              WR48-63:      32924 (     132.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9787875 (   39419.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     499592 (    2012.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      248.300 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.028               16.674
  1                4.476                0.000
  2                0.095                0.000
  3                4.784                0.000
  4                0.046               31.798
  5                8.905                0.000
  6                0.093                3.062
  7                8.121                0.000
  8                0.937                0.000
  9                0.000               59.989
 10                0.965                0.000
 11                0.025                0.000
 12                0.042                0.000
 13                0.000               63.078
 14                0.061                0.000
 15                0.217                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5702434       2900          0          0          3          3        703        703          0       1488          0      26833
  1   25200522       6341          0          0         31         30       3140       3140          0          0          0      55809
  2   10737118    2562866          0          0          3          3       1687       1687          0    2559486          0      44132
  3   32448674    2519917          0          0         37         37       4558       4558          0    2510727          0      63444
  4    3679860    1335111          0          0          5          5        577        577          0    1333947          0      11808
  5   37244520     759044          0          0         62         62       4940       4940          0     749040          0      34314
  6    9560033    1568630          0          0          3          3       1348       1348          0    1565928          0      23903
  7   40864979    1077667          0          0         66         66       5138       5138          0    1067259          0      40193
  8   17664728       4388          0          0          6          6       2188       2188          0          0          0      56741
  9    2360528        644          0          0          0          0        322        322          0          0          0       4285
 10   20213561       5020          0          0         12         12       2498       2498          0          0          0      34696
 11   11474187       2786          0          0          1          1       1392       1392          0          0          0      10298
 12    8797735       2188          0          0          1          1       1093       1093          0          0          0      49042
 13    2441760        604          0          0          0          0        302        302          0          0          0       3463
 14    7875814       1932          0          0          4          4        962        962          0          0          0      33004
 15   15799343       4168          0          0          8          8       2076       2076          0          0          0       7627
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.352s (  931.544 blocks/s,  58.222 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Bias2.png
Bias2.png
Attachment 7: Stats3.png
Stats3.png
Attachment 8: Bias3.png
Bias3.png
Attachment 9: Analysis_R14_371
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252182561 ( 1038396.9 Hz)
        Other data format:    9737439 (   40095.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        234 (       1.0 Hz)
                               RESUME:        234 (       1.0 Hz)
                              SYNC100:      32904 (     135.5 Hz)
                              WR48-63:      32904 (     135.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9671163 (   39822.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     493534 (    2032.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      242.858 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.060               15.876
  1                5.312                0.000
  2                0.091                0.000
  3                3.590                0.000
  4                0.163               29.634
  5                9.407                0.000
  6                0.049                0.000
  7                6.820                0.000
  8                2.387                0.000
  9                0.000               29.724
 10                0.676                0.000
 11                0.043                0.000
 12                0.046                0.000
 13                0.000               57.696
 14                0.016                3.102
 15                0.214                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5708701       2832          0          0          4          4        686        686          0       1452          0      26548
  1   24896226       6554          0          0         36         36       3241       3241          0          0          0      54251
  2   10734245    2512200          0          0          3          3       1683       1683          0    2508828          0      43373
  3   32202526    2481239          0          0         32         32       4391       4391          0    2472393          0      63125
  4    3649395    1322612          0          0         11         11        611        611          0    1321368          0      11610
  5   36874871     743171          0          0         64         64       4722       4722          0     733599          0      33565
  6    9551266    1557362          0          0          1          1       1434       1434          0    1554492          0      23558
  7   41439213    1089893          0          0         57         57       5374       5374          0    1079031          0      40478
  8   17460031       4238          0          0          7          7       2112       2112          0          0          0      55845
  9    2360192        582          0          0          0          0        291        291          0          0          0       4290
 10   20593873       5182          0          0          6          6       2585       2585          0          0          0      34440
 11   11697470       2942          0          0          2          2       1469       1469          0          0          0      10244
 12    8815559       2192          0          0          1          1       1095       1095          0          0          0      48502
 13    2454200        586          0          0          0          0        293        293          0          0          0       3481
 14    7942567       1924          0          0          2          2        960        960          0          0          0      32639
 15   15802226       3930          0          0          8          8       1957       1957          0          0          0       7585
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.879s (  944.541 blocks/s,  59.034 Mb/s)
Attachment 10: Stats4.png
Stats4.png
Attachment 11: Bias4.png
Bias4.png
  323   Tue May 18 06:50:53 2021 OH, PPTuesday 18th May 08:00 - 16:00
08:00 OH Takes over

      Statistics still look good from yesterday. Rate is still down.
      Dead time in FEEs 2 and 4 is 2.04%

      System wide checks clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x500 : 	 38  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      *Baseline reset*
      FPGA check ok
      Stats - attachment 1
      Temp - attachment 2
      Bias - attachment 3
      

08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
      I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
      The results of the test appear to indicate that it is the network link that we are saturating causing dead time
***	Before		                  After	
FEE	DT (s)  Dead time %	           DT (s)	Dead time %
0	0.013	0.004184451	           0.026	0.010025681
1	6.352	2.04458693	           4.097	1.57981599
2	0.295	0.09495484	           0.292	0.112596112
3	6.349	2.043621288	           8.255	3.183153771
4	0.021	0.006759497	           0.03	        0.011568094
5	0.285	0.091736032	           3.388	1.306423377
6	0.02	0.006437616	           0.149	0.057454865
7	1.219	0.392372712	           2.133	0.822491459
8	1.137	0.365978485	           0.457	0.176220627
9	0	0	                       0	0
10	1.633	0.525631369	           1.545	0.595756823
11	0.061	0.01963473	           0.166	0.064010118
12	0.059	0.018990968	           0.085	0.032776265
13	0	0	                       0	0
14	0.016	0.005150093	           0.023	0.008868872
15	0.282	0.09077039	           0.168	0.064781324


      Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased

09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
      They will now begin to change the spill structure.

09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change

10:21 Beam is back on AIDA

10:39 System wide checks. 
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x506 : 	 6  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

11:00 
      Stats - attachment 4
      Temp - attachment 5
      Bias - attachment 6

12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50b : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50f : 	 15  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed
Attachment 1: 210518_0757_Stats.png
210518_0757_Stats.png
Attachment 2: 210518_0758_Temp.png
210518_0758_Temp.png
Attachment 3: 210518_0759_Bias.png
210518_0759_Bias.png
Attachment 4: 210518_1059_Stats.png
210518_1059_Stats.png
Attachment 5: 210518_1100_Temp.png
210518_1100_Temp.png
Attachment 6: 210518_1101_Bias.png
210518_1101_Bias.png
  322   Mon May 17 23:17:16 2021 DSJTuesday 18 May 0.00-8.00 shift

0.00 - Taking over from Lewis 

00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2)

01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4)

01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6)

02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8)

Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4ee :      20  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     43     12      6      4      1      4      2      2      1      3      7   : 38444
aida02 :     21     12      4      4      2      3      3      3      3      3      6   : 36980
aida03 :     23      8      4      2      2      4      2      3      3      3      6   : 36764
aida04 :     34     12     10      5      1      4      1      4      3      3      6   : 37224
aida05 :     18      5      5      1      2      3      2      2      1      3      7   : 38112
aida06 :     40     12      5      7      2      4      2      4      3      3      6   : 37552
aida07 :     40     10      6      2      2      4      2      2      2      2      7   : 37392
aida08 :     19     11      3      3      1      4      3      3      3      3      6   : 36980
aida09 :     43      6      2      4      3      4      3      3      3      3      6   : 37180
aida10 :     32     14     10      3      4      1      2      4      1      3      7   : 39280
aida11 :     40      7      2      5      4      3      1      4      3      3      6   : 37144
aida12 :     37     14     13      2      2      3      2      4      3      3      6   : 37396
aida13 :     18      9      3      1      1      5      3      3      3      3      6   : 37024
aida14 :     36     13      6      4      2      3      2      3      1      3      7   : 38872
aida15 :     32     11      7      4      3      4      2      3      2      4      6   : 38024
aida16 :     26      9      4      1      0      3      1      4      3      3      6   : 36752

 

02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13)

03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15)

03.05 - No beam, problem with SIS, don't know how long will be off

03.14 - Beam is back

03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17)

04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19)

Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16


         Base         Current     Difference
aida05 fault      0x4da :      0x4f2 :      24  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     26     11      5      6      1      4      2      2      1      3      7   : 38416
aida02 :     16      9      5      4      2      4      2      2      3      3      6   : 36312
aida03 :      9      7      4      4      2      4      2      3      3      3      6   : 36764
aida04 :     36     11      8      4      3      4      1      3      4      3      6   : 37800
aida05 :     20      4      3      2      2      3      2      2      1      3      7   : 38112
aida06 :     22     13     10      7      2      4      3      3      4      3      6   : 38336
aida07 :     28      7      7      2      2      4      2      2      2      2      7   : 37336
aida08 :      2      3      4      4      1      3      2      3      3      3      6   : 36512
aida09 :     40      7      1      4      4      3      3      3      3      3      6   : 37096
aida10 :     26      9     12      2      3      2      2      4      1      3      7   : 39280
aida11 :     30      7      1      5      4      3      1      3      2      4      6   : 37600
aida12 :     15     10      6      4      2      3      2      4      3      3      6   : 37228
aida13 :      1      3      2      1      2      4      3      2      3      3      6   : 36316
aida14 :     34     12      7      4      2      3      2      3      1      3      7   : 38872
aida15 :     18     11      5      3      4      4      2      3      2      4      6   : 37968
aida16 :     22      7      4      2      0      3      1      4      3      3      6   : 36752

04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24)

05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26)

05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28)

06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30)

Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4f9 :      31  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     16      9      3      7      1      5      1      3      1      3      7   : 38744
aida02 :     18      3      5      2      3      3      3      2      3      3      6   : 36400
aida03 :     21      9      4      2      2      4      2      4      3      3      6   : 37276
aida04 :     32     12      7      3      3      4      1      3      4      3      6   : 37744
aida05 :     12      8      3      2      2      4      2      2      1      3      7   : 38240
aida06 :     28     14      4      8      2      4      3      3      3      3      6   : 37280
aida07 :     40     12      5      2      2      4      1      3      2      2      7   : 37648
aida08 :     13      8      4      4      1      3      2      3      3      3      6   : 36596
aida09 :     35      5      2      4      3      4      2      4      3      3      6   : 37396
aida10 :      6     10      4      1      1      3      2      2      1      3      7   : 38024
aida11 :     41      9      1      4      4      3      2      4      3      3      6   : 37372
aida12 :     29     11      9      4      2      3      2      4      3      3      6   : 37340
aida13 :     24     10      5      1      2      4      3      2      3      3      6   : 36512
aida14 :     34     16      5      4      2      3      2      3      1      3      7   : 38872
aida15 :     42      8      6      2      4      4      2      3      2      4      6   : 38024
aida16 :      8      9      6      1      0      3      1      4      3      3      6   : 36712

06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35)

07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37)

07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39)

Attachment 1: Screenshot_2021-05-18_Statistics_aidas-gsi(1).png
Screenshot_2021-05-18_Statistics_aidas-gsi(1).png
Attachment 2: R14_147_deadtime_23.41.06.png
R14_147_deadtime_23.41.06.png
Attachment 3: Screenshot_2021-05-18_Statistics_aidas-gsi(2).png
Screenshot_2021-05-18_Statistics_aidas-gsi(2).png
Attachment 4: R14_153_deadtime_00.07.12.png
R14_153_deadtime_00.07.12.png
Attachment 5: Screenshot_2021-05-18_Statistics_aidas-gsi(3).png
Screenshot_2021-05-18_Statistics_aidas-gsi(3).png
Attachment 6: R14_158_deadtime_00.32.42.png
R14_158_deadtime_00.32.42.png
Attachment 7: Screenshot_2021-05-18_Statistics_aidas-gsi(4).png
Screenshot_2021-05-18_Statistics_aidas-gsi(4).png
Attachment 8: R14_164_deadtime_01.02.32.png
R14_164_deadtime_01.02.32.png
Attachment 9: TMP.png
TMP.png
Attachment 10: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 11: Screenshot_2021-05-18_ucesb.png
Screenshot_2021-05-18_ucesb.png
Attachment 12: Screenshot_2021-05-18_Statistics_aidas-gsi(5).png
Screenshot_2021-05-18_Statistics_aidas-gsi(5).png
Attachment 13: R14_170_deadtime_01.35.24.png
R14_170_deadtime_01.35.24.png
Attachment 14: Screenshot_2021-05-18_Statistics_aidas-gsi(6).png
Screenshot_2021-05-18_Statistics_aidas-gsi(6).png
Attachment 15: R14_175_deadtime_01.59.37.png
R14_175_deadtime_01.59.37.png
Attachment 16: Screenshot_2021-05-18_Statistics_aidas-gsi(7).png
Screenshot_2021-05-18_Statistics_aidas-gsi(7).png
Attachment 17: R14_180_deadtime_02.31.32.png
R14_180_deadtime_02.31.32.png
Attachment 18: Screenshot_2021-05-18_Statistics_aidas-gsi(8).png
Screenshot_2021-05-18_Statistics_aidas-gsi(8).png
Attachment 19: R14_186_deadtime_03.00.27.png
R14_186_deadtime_03.00.27.png
Attachment 20: TMP.png
TMP.png
Attachment 21: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 22: Screenshot_2021-05-18_ucesb(1).png
Screenshot_2021-05-18_ucesb(1).png
Attachment 23: Screenshot_2021-05-18_Statistics_aidas-gsi(9).png
Screenshot_2021-05-18_Statistics_aidas-gsi(9).png
Attachment 24: R14_192_deadtime_03.31.26.png
R14_192_deadtime_03.31.26.png
Attachment 25: Screenshot_2021-05-18_Statistics_aidas-gsi(10).png
Screenshot_2021-05-18_Statistics_aidas-gsi(10).png
Attachment 26: R14_198_deadtime_04.00.21.png
R14_198_deadtime_04.00.21.png
Attachment 27: Screenshot_2021-05-18_Statistics_aidas-gsi(11).png
Screenshot_2021-05-18_Statistics_aidas-gsi(11).png
Attachment 28: R14_198_deadtime_04.27.46.png
R14_198_deadtime_04.27.46.png
Attachment 29: Screenshot_2021-05-18_Statistics_aidas-gsi(12).png
Screenshot_2021-05-18_Statistics_aidas-gsi(12).png
Attachment 30: R14_203_deadtime_05.png
R14_203_deadtime_05.png
Attachment 31: TMP.png
TMP.png
Attachment 32: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 33: Screenshot_2021-05-18_ucesb(3).png
Screenshot_2021-05-18_ucesb(3).png
Attachment 34: Screenshot_2021-05-18_Statistics_aidas-gsi(13).png
Screenshot_2021-05-18_Statistics_aidas-gsi(13).png
Attachment 35: R14_215_deadtime_05.30.26.png
R14_215_deadtime_05.30.26.png
Attachment 36: Screenshot_2021-05-18_Statistics_aidas-gsi(14).png
Screenshot_2021-05-18_Statistics_aidas-gsi(14).png
Attachment 37: R14__221_DEADTIME_06.00.35.png
R14__221_DEADTIME_06.00.35.png
Attachment 38: Screenshot_2021-05-18_Statistics_aidas-gsi(15).png
Screenshot_2021-05-18_Statistics_aidas-gsi(15).png
Attachment 39: R14_226_deadtime_06.25.32.png
R14_226_deadtime_06.25.32.png
  321   Mon May 17 19:05:23 2021 LSMonday 17th May 20.00-24.00
20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s

21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)

21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)

22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4e7 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.4M items/s
Tape server 6.3MB/s

Ran R14_119 through analyser, dead times all good (Fig 11)

22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)

23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)

23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)

24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4eb : 	 17  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.5M items/s
Tape server 6.5MB/s

Ran R14_142 through analyser, dead times all good (Fig 21)
Attachment 1: 210517_Stats_2026.png
210517_Stats_2026.png
Attachment 2: 210517_Deadtime_R14_99.png
210517_Deadtime_R14_99.png
Attachment 3: 210517_Stats_2101.png
210517_Stats_2101.png
Attachment 4: 210517_Deadtimes_R14_106.png
210517_Deadtimes_R14_106.png
Attachment 5: 210517_Stats_2128.png
210517_Stats_2128.png
Attachment 6: 210517_Deadtimes_R14_112.png
210517_Deadtimes_R14_112.png
Attachment 7: 210517_Stats_2159.png
210517_Stats_2159.png
Attachment 8: 210517_FEEtemps_2159.png
210517_FEEtemps_2159.png
Attachment 9: 210517_Spectra_2159.png
210517_Spectra_2159.png
Attachment 10: 210517_Bias_2159.png
210517_Bias_2159.png
Attachment 11: 210517_Deadtimes_R14_119.png
210517_Deadtimes_R14_119.png
Attachment 12: 210517_Stats_2230.png
210517_Stats_2230.png
Attachment 13: 210517_Deadtimes_R14_123.png
210517_Deadtimes_R14_123.png
Attachment 14: 210517_Stats_2300.png
210517_Stats_2300.png
Attachment 15: 210517_Deadtimes_R14_129.png
210517_Deadtimes_R14_129.png
Attachment 16: 210517_Stats_2328.png
210517_Stats_2328.png
Attachment 17: 210517_Deadtimes_R14_135.png
210517_Deadtimes_R14_135.png
Attachment 18: 210517_Stats_2359.png
210517_Stats_2359.png
Attachment 19: 210517_FEEtemps_2359.png
210517_FEEtemps_2359.png
Attachment 20: 210517_bias_2359.png
210517_bias_2359.png
Attachment 21: 210517_deadtime_R14_142_1.png
210517_deadtime_R14_142_1.png
  320   Mon May 17 15:14:35 2021 LJWMonday 17th May 2021 16:00-20:00

16:25

Informed at shift handover that there has been no beam.

Statistics  - See attachment 1

Temperatures - See attachment 2

Bias & Leakage currents - See attachment 3

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit decoder status:

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4dd :      3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

18:00

Usecb scalars now showing DSSD implant rates.

Statistics  - See attachment 4

Temperatures - See attachment 5

Bias & Leakage currents - See attachment 6

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx

 

19:45

Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.

Statistics  - See attachment 7

Temperatures - See attachment 8

Bias & Leakage currents - See attachment 9

System Wide checks - All passed, EXCEPT:

ADC Calibration:

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit decoder status:

     Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes

19:55 - Beam now back again - DSSD implants showing

 

 

Attachment 1: 170521_1.png
170521_1.png
Attachment 2: 170521_2.png
170521_2.png
Attachment 3: 170521_3.png
170521_3.png
Attachment 4: 170521_4.png
170521_4.png
Attachment 5: 170521_5.png
170521_5.png
Attachment 6: 170521_6.png
170521_6.png
Attachment 7: 170521_7.png
170521_7.png
Attachment 8: 170521_8.png
170521_8.png
Attachment 9: 170521_9.png
170521_9.png
  319   Fri May 14 19:01:09 2021 JSFriday 14th May 20:00-00:00
20:00 Taking over from CB

20:33 Doing full checks.
Temps ok elog:319/8  
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

FPGA Timestamp errors -
		 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     34     30      8      1      4      1      2      2      3      6   : 35772
aida02 :     37     30     28      5      3      1      2      4      2      3      6   : 36644
aida03 :     41     29     30     16      4      3      1      5      1      3      6   : 36588
aida04 :     13      6     23     12      4      4      4      4      2      2      6   : 35412
aida05 :     20     32     23     10      2      2      2      2      1      4      6   : 36736
aida06 :     14      5     10      5      1      3      2      3      2      3      6   : 35680
aida07 :     44     23     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     39     30     28      9      0      3      2      3      3      2      6   : 35308
aida09 :     41     31     28     10      2      5      1      3      1      3      6   : 35484
aida10 :     43     23     27      7      3      1      1      3      3      3      6   : 36916
aida11 :     39     35     17     10      3      3      2      2      2      3      6   : 35908
aida12 :     37     36     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     47     32     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     23     33     14     10      3      4      3      4      3      2      6   : 36164
aida15 :     44     28     24     12      3      2      4      3      3      2      6   : 35920
aida16 :     22     17     24     17      3      4      2      3      1      3      6   : 35648

Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27 
Resume info #3 - elog:319/3 same as elog:317/26 
Pause info #2 - elog:319/4 same as elog:317/32 
AIDA disk info #6 - elog:319/5 same as elog:317/31 
AIDA ADC data items - elog:319/6 same as elog:317/30 
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see 
clearly once uploaded the screenshots] 

Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%

21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay

21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     39     34     29      8      1      4      1      2      2      3      6   : 35772
aida02 :     46     29     28      5      3      2      2      3      2      3      6   : 36288
aida03 :     39     25     29     16      4      2      2      5      1      3      6   : 36660
aida04 :     14      6     19     12      4      4      3      4      2      2      6   : 35096
aida05 :     20     30     24     10      2      2      2      2      1      4      6   : 36736
aida06 :     26      4     11      5      2      4      3      2      2      3      6   : 35672
aida07 :     46     22     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     28     32     28      9      0      2      2      4      3      2      6   : 35664
aida09 :     32     30     29     10      3      5      2      3      1      3      6   : 35776
aida10 :     39     23     25      7      3      1      1      3      3      3      6   : 36868
aida11 :     40     31     21     10      3      3      1      4      1      3      6   : 35688
aida12 :     17     31     12     10      2      2      3      4      3      2      6   : 35772
aida13 :     35     28     22      7      4      2      3      4      3      2      6   : 36012
aida14 :     30     33     14     10      3      4      3      4      3      2      6   : 36192
aida15 :     34     26     24     12      3      2      4      3      3      2      6   : 35864
aida16 :     38     24     27     17      3      4      2      3      1      3      6   : 35816

Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12 
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14 
Aida Correlation Info #8 - elog:319/15  - rates showed zero, checked again and normal elog:319/17

Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%

22:00
Temps ok
Stats ok 
ucesb ok

22:35 
Temps ok
Stats ok 
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up 
again elog:319/17

22:54 They are closing the file, we are on R5_155

23:00 
Attachment 1: bias1.png
bias1.png
Attachment 2: Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Attachment 3: Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Attachment 4: Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Attachment 5: Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Attachment 7: Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Attachment 8: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: R5_108_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256861958 ( 1503007.9 Hz)
        Other data format:    5058042 (   29596.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        150 (       0.9 Hz)
                              SYNC100:      32636 (     191.0 Hz)
                              WR48-63:      32636 (     191.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4992471 (   29213.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     207149 (    1212.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.899 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                4.459               72.353
  2                1.217               86.117
  3                3.357               28.177
  4                0.000              160.970
  5                0.495               76.386
  6                0.216               93.617
  7                1.418               40.259
  8                3.253               44.571
  9                0.000              168.837
 10                5.250                9.058
 11                1.956               49.014
 12                0.182               82.483
 13                0.000              160.585
 14                0.143               88.740
 15                3.053               35.456
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8464368       3303          0          0          2          2       1138       1138          0       1023          0      11424
  1   16399642       4233          0          0         17         18       2099       2099          0          0          0      22505
  2   13045073    1576129          0          0          6          6       1793       1793          0    1572531          0      18179
  3   26563169    1558422          0          0         14         14       3487       3487          0    1551420          0      26353
  4    2213090     510296          0          0          0          0        328        328          0     509640          0       5100
  5   15498485     314714          0          0          4          4       1986       1986          0     310734          0      15610
  6   12553775     609520          0          0          7          7       1573       1573          0     606360          0      10360
  7   23384876     446841          0          0         10         10       3029       3029          0     440763          0      18175
  8   21832531       5542          0          0         15         15       2756       2756          0          0          0      22662
  9    1045474        278          0          0          0          0        139        139          0          0          0       1866
 10   42357744      10552          0          0         43         43       5233       5233          0          0          0      12639
 11   20606319       4998          0          0         11         11       2488       2488          0          0          0       4351
 12   14311836       3576          0          0          6          6       1782       1782          0          0          0      20348
 13    3016503        780          0          0          0          0        390        390          0          0          0       1508
 14   13047482       3248          0          0          4          4       1620       1620          0          0          0      12826
 15   22521591       5610          0          0         10         10       2795       2795          0          0          0       3243
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:74697.445s (    0.428 blocks/s,   0.027 Mb/s)
Attachment 10: Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Attachment 11: Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Attachment 12: Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Attachment 16: R5_128_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256841478 ( 1482693.9 Hz)
        Other data format:    5078522 (   29317.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        167 (       1.0 Hz)
                               RESUME:        166 (       1.0 Hz)
                              SYNC100:      32644 (     188.4 Hz)
                              WR48-63:      32644 (     188.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5012901 (   28938.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     205165 (    1184.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      173.226 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.621               76.080
  2                1.556               81.876
  3                3.719               27.628
  4                0.000              162.954
  5                0.710               80.266
  6                0.170               93.769
  7                2.714               44.740
  8                3.483               45.565
  9                0.000              167.005
 10                8.522               10.948
 11                2.181               50.562
 12                0.202               85.035
 13                0.000              161.510
 14                0.146               88.959
 15                1.812               38.597
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8468806       3273          0          0          7          7       1112       1112          0       1035          0      11260
  1   16364560       4360          0          0         13         13       2167       2167          0          0          0      23505
  2   13265609    1594590          0          0          6          6       1881       1881          0    1590816          0      18254
  3   26054686    1576337          0          0         14         14       3422       3422          0    1569465          0      26152
  4    2223261     507850          0          0          0          0        362        362          0     507126          0       4952
  5   15152054     312213          0          0          8          7       1938       1938          0     308322          0      15336
  6   12659379     606961          0          0          4          4       1612       1612          0     603729          0      10099
  7   22669666     438118          0          0         14         14       2841       2841          0     432408          0      17594
  8   22337729       5564          0          0         15         15       2767       2767          0          0          0      22524
  9    1020352        240          0          0          0          0        120        120          0          0          0       1807
 10   42596491      10676          0          0         59         59       5279       5279          0          0          0      12076
 11   20672887       5060          0          0         13         13       2517       2517          0          0          0       4379
 12   14335521       3608          0          0          5          5       1799       1799          0          0          0      20105
 13    2812573        690          0          0          0          0        345        345          0          0          0       1405
 14   13325633       3260          0          0          4          4       1626       1626          0          0          0      12554
 15   22882271       5722          0          0          5          5       2856       2856          0          0          0       3163
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:78055.344s (    0.410 blocks/s,   0.026 Mb/s)
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Attachment 18: ucesb.png
ucesb.png
Attachment 19: _~.png
_~.png
Attachment 20: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 21: Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Attachment 22: Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Attachment 23: Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Attachment 24: Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Attachment 25: Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Attachment 26: Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Attachment 27: R5_157_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256650793 ( 1485751.4 Hz)
        Other data format:    5269207 (   30503.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        130 (       0.8 Hz)
                               RESUME:        129 (       0.7 Hz)
                              SYNC100:      32643 (     189.0 Hz)
                              WR48-63:      32643 (     189.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5203662 (   30124.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     223116 (    1291.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      172.741 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.094               74.418
  2                0.927               76.576
  3                3.655               26.911
  4                0.000              161.752
  5                0.023               80.653
  6                0.215               91.583
  7                1.871               44.002
  8                1.038               39.676
  9                0.000              166.718
 10                8.591               10.747
 11                3.917               45.280
 12                0.214               88.240
 13                0.000              163.408
 14                0.280               88.718
 15                1.416               44.122
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8604841       3308          0          0          0          0       1138       1138          0       1032          0      12424
  1   16359820       4228          0          0         13         13       2101       2101          0          0          0      24974
  2   13547296    1610585          0          0          6          6       1930       1930          0    1606713          0      19730
  3   25981004    1589167          0          0         14         13       3491       3491          0    1582158          0      28914
  4    2316430     549243          0          0          0          0        351        351          0     548541          0       5271
  5   15208801     341870          0          0          1          1       1974       1974          0     337920          0      16457
  6   12807457     657341          0          0          5          5       1667       1667          0     653997          0      10925
  7   22697274     479019          0          0         10         10       2849       2849          0     473301          0      19398
  8   22587880       5594          0          0          5          5       2792       2792          0          0          0      24704
  9    1078064        266          0          0          0          0        133        133          0          0          0       1963
 10   42543713      10740          0          0         43         43       5327       5327          0          0          0      13313
 11   20362862       5182          0          0         14         14       2577       2577          0          0          0       4606
 12   13994455       3436          0          0          7          7       1711       1711          0          0          0      21818
 13    2731986        666          0          0          0          0        333        333          0          0          0       1621
 14   13145005       3222          0          0          8          8       1603       1603          0          0          0      13539
 15   22683905       5340          0          0          4          4       2666       2666          0          0          0       3459
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:83137.008s (    0.385 blocks/s,   0.024 Mb/s)
  318   Fri May 14 17:30:56 2021 TDAnalysis of S496 files R5_18, R5_32 & R5_45
S496/R5_18 - attachment 1

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 0xf
                             n+n FEE64s 0x20
Sampling ADCs OFF


S496/R5_32 - attachment 2

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 & aida15 0xf
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


S496/R5_45 - attachment 3

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xc
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


* N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
 
Attachment 1: R5_18
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260127700 ( 2662102.7 Hz)
        Other data format:    1792300 (   18342.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1843 (      18.9 Hz)
                               RESUME:       1843 (      18.9 Hz)
                              SYNC100:      32365 (     331.2 Hz)
                              WR48-63:      32365 (     331.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1723884 (   17641.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      65165 (     666.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       97.715 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1               36.653               41.693
  2               44.272               48.364
  3               46.942               50.228
  4                0.000               81.057
  5               27.042               29.539
  6               41.734               45.821
  7               43.168               46.980
  8               54.980               58.850
  9                4.477               17.487
 10               57.700               61.776
 11               43.911               47.867
 12               48.996               52.920
 13               21.829               26.853
 14               51.720               56.077
 15               53.092               56.077
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16755449       4921          0          0        122        122       2151       2151          0        375          0       4078
  1   15729031       4184          0          0        103        103       1989       1989          0          0          0       7687
  2   17390463     502955          0          0        136        135       2294       2294          0     498096          0       5652
  3   14745088     476937          0          0        118        118       1907       1907          0     472887          0       8350
  4    2857057     295831          0          0          0          0        413        413          0     295005          0       2971
  5   19341027     128907          0          0        112        112       2426       2426          0     123831          0       5913
  6   17649809     199902          0          0        136        136       2168       2168          0     195294          0       3189
  7   17833639     143068          0          0        135        135       2201       2201          0     138396          0       5626
  8   17514114       4531          0          0        139        138       2127       2127          0          0          0       6021
  9   14922420       3686          0          0         23         23       1820       1820          0          0          0       1039
 10   16909910       4387          0          0        140        141       2053       2053          0          0          0       3196
 11   18661933       4896          0          0        151        151       2297       2297          0          0          0       1392
 12   17768788       4718          0          0        148        148       2211       2211          0          0          0       5274
 13   16872590       4258          0          0         96         96       2033       2033          0          0          0        605
 14   16747047       4335          0          0        134        135       2033       2033          0          0          0       3273
 15   18429335       4784          0          0        150        150       2242       2242          0          0          0        899
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:66231.297s (    0.483 blocks/s,   0.030 Mb/s)
Attachment 2: R5_32
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259165945 ( 2447522.6 Hz)
        Other data format:    2754055 (   26008.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1068 (      10.1 Hz)
                               RESUME:       1070 (      10.1 Hz)
                              SYNC100:      32395 (     305.9 Hz)
                              WR48-63:      32395 (     305.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2687127 (   25376.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     109002 (    1029.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      105.889 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.772               22.665
  2               28.336               32.781
  3                9.177               18.264
  4                0.000               94.253
  5                5.882               26.886
  6                8.897               16.463
  7               18.512               26.355
  8               29.893               33.327
  9                0.057               79.535
 10               45.010               46.522
 11               27.988               27.766
 12               32.050               37.161
 13                1.981               23.343
 14                0.002               96.477
 15               36.327               39.608
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17728100       5332          0          0         32         32       2340       2340          0        588          0       6827
  1   15060064       3752          0          0         18         18       1858       1858          0          0          0      14360
  2   18674317     723732          0          0         99         99       2382       2382          0     718770          0       8046
  3   19118104     904863          0          0         39         39       2463       2463          0     899859          0      15890
  4    2260998     321658          0          0          0          0        326        326          0     321006          0       3138
  5   14506811     189277          0          0         25         25       1756       1756          0     185715          0       9208
  6   21798062     340177          0          0         43         43       2689       2689          0     334713          0       5553
  7   18561150     231264          0          0         70         70       2324       2324          0     226476          0       9337
  8   22060606       5804          0          0        136        136       2766       2766          0          0          0      10232
  9    5139884       1246          0          0          5          5        618        618          0          0          0       1276
 10   24215101       6255          0          0        190        191       2937       2937          0          0          0       4375
 11   22485598       5956          0          0        148        148       2830       2830          0          0          0       2034
 12   19527530       4883          0          0        108        109       2333       2333          0          0          0       8243
 13   14237341       3538          0          0          9          9       1760       1760          0          0          0        984
 14    2260329        600          0          0          1          1        299        299          0          0          0       8235
 15   21531950       5718          0          0        145        145       2714       2714          0          0          0       1264
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:65849.211s (    0.486 blocks/s,   0.030 Mb/s)
Attachment 3: R5_45
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
  317   Fri May 14 14:57:32 2021 CB + OH, TD, NHFriday 14th May 16:00-20:00
16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S
Attachment 1: Temp1.png
Temp1.png
Attachment 2: Bias1.png
Bias1.png
Attachment 3: Stats1-1.png
Stats1-1.png
Attachment 4: Stats1-2.png
Stats1-2.png
Attachment 5: Stats1-3.png
Stats1-3.png
Attachment 6: Stats1-4.png
Stats1-4.png
Attachment 7: Stats1-5.png
Stats1-5.png
Attachment 8: Stats1-6.png
Stats1-6.png
Attachment 9: R5_45_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
Attachment 10: Temps2.png
Temps2.png
Attachment 11: Bias2.png
Bias2.png
Attachment 12: Stats2-1.png
Stats2-1.png
Attachment 13: Stats2-2.png
Stats2-2.png
Attachment 14: Stats2-3.png
Stats2-3.png
Attachment 15: Stats2-4.png
Stats2-4.png
Attachment 16: Stats2-5.png
Stats2-5.png
Attachment 17: Stats2-6.png
Stats2-6.png
Attachment 18: Temp3.png
Temp3.png
Attachment 19: Bias3.png
Bias3.png
Attachment 20: Stats3-1.png
Stats3-1.png
Attachment 21: Stats3-2.png
Stats3-2.png
Attachment 22: Stats3-3.png
Stats3-3.png
Attachment 23: Stats3-4.png
Stats3-4.png
Attachment 24: Stats3-5.png
Stats3-5.png
Attachment 25: R5_67_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255938817 ( 1305497.2 Hz)
        Other data format:    5981183 (   30508.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.6 Hz)
                               RESUME:        113 (       0.6 Hz)
                              SYNC100:      32730 (     166.9 Hz)
                              WR48-63:      32730 (     166.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5915496 (   30173.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     257806 (    1315.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      196.047 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.111              102.399
  2                0.513              105.076
  3                3.587               39.451
  4                0.000              182.105
  5                0.792              103.780
  6                0.167              115.991
  7                1.244               72.680
  8                0.805               56.828
  9                0.000              190.127
 10                2.629                6.967
 11                0.295               76.751
 12                0.222              114.996
 13                0.000              188.225
 14                0.031              124.690
 15                2.491               63.947
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8223770       3364          0          0          2          2       1092       1092          0       1176          0      13825
  1   16628095       4338          0          0         12         12       2157       2157          0          0          0      28636
  2   13173045    1831289          0          0          3          3       1864       1864          0    1827555          0      22667
  3   27251797    1807380          0          0         13         13       3668       3668          0    1800018          0      33190
  4    2564061     622927          0          0          0          0        404        404          0     622119          0       6108
  5   14810679     385418          0          0          6          6       1903       1903          0     381600          0      19053
  6   12067625     742516          0          0          6          6       1616       1616          0     739272          0      12600
  7   21266971     549224          0          0         11         11       2723       2723          0     543756          0      22659
  8   23290358       5786          0          0          5          5       2888       2888          0          0          0      28199
  9    1157906        324          0          0          0          0        162        162          0          0          0       2443
 10   45608185      11489          0          0         27         26       5718       5718          0          0          0      16139
 11   20146427       4876          0          0          6          6       2432       2432          0          0          0       5579
 12   13411417       3300          0          0          7          7       1643       1643          0          0          0      24922
 13    2681820        654          0          0          0          0        327        327          0          0          0       1925
 14   11884997       2904          0          0          1          1       1451       1451          0          0          0      15892
 15   21771664       5394          0          0         15         15       2682       2682          0          0          0       3969
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:67311.180s (    0.475 blocks/s,   0.030 Mb/s)
Attachment 26: Stats_RESUME.png
Stats_RESUME.png
Attachment 27: Stats-CORRELATION.png
Stats-CORRELATION.png
Attachment 28: Bias4.png
Bias4.png
Attachment 29: Stats-good.png
Stats-good.png
Attachment 30: Stats-AIDAADC.png
Stats-AIDAADC.png
Attachment 31: Stats_DISC.png
Stats_DISC.png
Attachment 32: Stats_PAUSE.png
Stats_PAUSE.png
Attachment 33: R5_86_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256871525 ( 1549964.4 Hz)
        Other data format:    5048475 (   30462.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        133 (       0.8 Hz)
                               RESUME:        132 (       0.8 Hz)
                              SYNC100:      32617 (     196.8 Hz)
                              WR48-63:      32617 (     196.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4982976 (   30067.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     217412 (    1311.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      165.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.819               66.425
  2                0.365               78.785
  3                3.792               29.131
  4                0.000              149.995
  5                0.868               74.751
  6                0.080               85.583
  7                2.138               33.988
  8                2.899               39.410
  9                0.000              164.566
 10                6.499               11.006
 11                1.139               43.564
 12                0.944               80.601
 13                0.000              152.925
 14                0.102               87.597
 15                2.546               36.680
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8622288       3316          0          0          4          3       1158       1158          0        993          0      11812
  1   17129008       4510          0          0         12         12       2243       2243          0          0          0      24429
  2   13474006    1553973          0          0          6          6       1869       1869          0    1550223          0      19028
  3   26129234    1524704          0          0         20         20       3374       3374          0    1517916          0      27835
  4    2168475     521045          0          0          0          0        352        352          0     520341          0       5247
  5   15823578     326080          0          0          4          4       1963       1963          0     322146          0      16074
  6   12515224     622754          0          0          2          2       1625       1625          0     619500          0      10516
  7   23990039     458079          0          0         12         12       3099       3099          0     451857          0      18924
  8   22108276       5660          0          0         13         13       2817       2817          0          0          0      24206
  9    1062355        236          0          0          0          0        118        118          0          0          0       1949
 10   41223894      10244          0          0         40         40       5082       5082          0          0          0      13136
 11   20572413       4974          0          0          7          7       2480       2480          0          0          0       4577
 12   14036482       3446          0          0          3          3       1720       1720          0          0          0      21370
 13    3191210        818          0          0          0          0        409        409          0          0          0       1573
 14   12949121       3198          0          0          3          3       1596       1596          0          0          0      13450
 15   21875922       5438          0          0          7          7       2712       2712          0          0          0       3286
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:70812.570s (    0.452 blocks/s,   0.028 Mb/s)
  316   Fri May 14 07:05:08 2021 CAFriday 14th May 08:00 - 16:00 shift
08:00 CA takes over

08:28 Problem at UNILAC - no beam

09:30 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveforms)

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA timestamp errors:

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:36 FEE64 Temperatures ok - attachment 1

      Detector bias / leakage currents ok - attachment 2

      Statistics - attachments 3-8

09:50 beam is back - writing to file R4_265

09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA

10:10 FRS increase degrader thickness

10.28 all histograms, stats and merger stats zero'd

10.50 all system wide checks ok *except*
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      2      2      2      2      2      2      4      3      3      6   : 36860
aida02 :      7      3      5      1      2      5      2      4      3      3      6   : 37284
aida03 :      7      5      3      1      4      4      2      3      3      3      6   : 36756
aida04 :      3      1     20      4      2      3      3      5      1      3      6   : 36052
aida05 :     19     25     26      6      2      2      3      2      2      3      6   : 35828
aida06 :      1      3     16      1      3      4      1      5      1      3      6   : 35580
aida07 :      6      4      1      2      3      3      2      3      3      3      6   : 36552
aida08 :      8      5      2      3      2      3      2      2      2      4      6   : 37064
aida09 :      9      7      1      2      2      4      2      3      3      3      6   : 36652
aida10 :      2      5      3      1      1      4      2      3      3      3      6   : 36544
aida11 :     16      4      4      2      1      2      2      3      3      3      6   : 36384
aida12 :      2      1      1      3      1      2      2      3      3      3      6   : 36288
aida13 :      0      1      1      2      0      4      3      2      3      3      6   : 36184
aida14 :      6      3      2      0      2      3      2      3      3      3      6   : 36432
aida15 :     25      8      2      1      2      3      1      2      2      4      6   : 36836
aida16 :      3      5      1      2      2      2      3      2      3      3      6   : 36100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.50 DAQ continues S496/R4_304

11:18 FEE64 Temperatures ok - attachment 9

      Detector bias & leakage currents ok - attachment 10

      statistics - attachments 11-16

15:00 FEE64           avg. CPU usage (%)

      1               52
      2               55
      3               55
      4               60
      5               93
      6               50
      7               53
      8               55
      9               55
      10              51
      11              50
      12              67
      13              56
      14              55
      15              55
      16              58

15:08 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveform)

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA errors:

 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:10 Temperatures ok - attachment 17

      Detector bias / leakage current ok - attachment 18

      Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png
Screenshot_from_2021-05-14_08-32-16.png
Attachment 2: Screenshot_from_2021-05-14_08-32-45.png
Screenshot_from_2021-05-14_08-32-45.png
Attachment 3: Screenshot_from_2021-05-14_08-33-11.png
Screenshot_from_2021-05-14_08-33-11.png
Attachment 4: Screenshot_from_2021-05-14_08-33-33.png
Screenshot_from_2021-05-14_08-33-33.png
Attachment 5: Screenshot_from_2021-05-14_08-34-12.png
Screenshot_from_2021-05-14_08-34-12.png
Attachment 6: Screenshot_from_2021-05-14_08-34-44.png
Screenshot_from_2021-05-14_08-34-44.png
Attachment 7: Screenshot_from_2021-05-14_08-35-17.png
Screenshot_from_2021-05-14_08-35-17.png
Attachment 8: Screenshot_from_2021-05-14_08-35-55.png
Screenshot_from_2021-05-14_08-35-55.png
Attachment 9: Screenshot_from_2021-05-14_10-17-25.png
Screenshot_from_2021-05-14_10-17-25.png
Attachment 10: Screenshot_from_2021-05-14_10-17-43.png
Screenshot_from_2021-05-14_10-17-43.png
Attachment 11: Screenshot_from_2021-05-14_10-22-39.png
Screenshot_from_2021-05-14_10-22-39.png
Attachment 12: Screenshot_from_2021-05-14_10-23-13.png
Screenshot_from_2021-05-14_10-23-13.png
Attachment 13: Screenshot_from_2021-05-14_10-23-41.png
Screenshot_from_2021-05-14_10-23-41.png
Attachment 14: Screenshot_from_2021-05-14_10-24-12.png
Screenshot_from_2021-05-14_10-24-12.png
Attachment 15: Screenshot_from_2021-05-14_10-24-37.png
Screenshot_from_2021-05-14_10-24-37.png
Attachment 16: Screenshot_from_2021-05-14_10-25-20.png
Screenshot_from_2021-05-14_10-25-20.png
Attachment 17: Screenshot_from_2021-05-14_14-07-48.png
Screenshot_from_2021-05-14_14-07-48.png
Attachment 18: Screenshot_from_2021-05-14_14-11-23.png
Screenshot_from_2021-05-14_14-11-23.png
Attachment 19: Screenshot_from_2021-05-14_14-13-21.png
Screenshot_from_2021-05-14_14-13-21.png
Attachment 20: Screenshot_from_2021-05-14_14-13-38.png
Screenshot_from_2021-05-14_14-13-38.png
Attachment 21: Screenshot_from_2021-05-14_14-14-12.png
Screenshot_from_2021-05-14_14-14-12.png
Attachment 22: Screenshot_from_2021-05-14_14-14-39.png
Screenshot_from_2021-05-14_14-14-39.png
Attachment 23: Screenshot_from_2021-05-14_14-15-06.png
Screenshot_from_2021-05-14_14-15-06.png
Attachment 24: Screenshot_from_2021-05-14_14-15-26.png
Screenshot_from_2021-05-14_14-15-26.png
  315   Thu May 13 23:58:05 2021 OHFriday 14th May 00:00 - 08:00
00:00 OH Takes over.
 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
0x210 

 p+n Thresholds 0xa (100keV)
 n+n Threshold 0x20 (320keV)

01:00 System wide checks
 Clock status ok
 ADC calibration fail as expected (Waveforms off)
 WR Errors
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dbf : 	 7  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 FPGA Errors
	
                Base Current Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Statistics ok - attachment 1
 Temp ok - attachment 2
 Bias and leakage current ok - attachment 3

01:42 Comparing the CS of AIDA3 and 4
      Stats have been running for a while (Believe since 5am)
      Looking at the counter ratio - attachment 4
      We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time)

01:56 Noticed /media/ThirdDrive was at 97% full
      Let DESPEC locals know I would like to use the next run stop to change the directory.
      They said they could do it now so they stopped their run.
      Changed the symbolic link to /media/1e2....
      Now writing to R4_

03:26 MBS DAQ Stopped as beam stopped
03:33 Beam back now and DAQ running again

04:44 System wide checks as before
      Statistics ok - attachment 5
      Temperatures ok - attachment - 7
      Bias and leakage currents ok - attachment 6

05:53 MBS stops run as beam drops
06:00 MBS started a new run as beam is back

07:37 System wide checks clock ok and FPGA as before
      WR additional difference on FEE5
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc0 : 	 8  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics ok - attachment 8
     Temp ok - attachment 9
     Bias and leakage ok - attachment 10
 
Attachment 1: 210514_0100_Stats.png
210514_0100_Stats.png
Attachment 2: 210514_0101_Temp.png
210514_0101_Temp.png
Attachment 3: 210514_0102_Bias.png
210514_0102_Bias.png
Attachment 4: 210514_0141_CS_Stats.png
210514_0141_CS_Stats.png
Attachment 5: 210514_0442_Stats.png
210514_0442_Stats.png
Attachment 6: 210514_0443_Bias.png
210514_0443_Bias.png
Attachment 7: 210514_0443_Temp.png
210514_0443_Temp.png
Attachment 8: 210514_0737_Stats.png
210514_0737_Stats.png
Attachment 9: 210514_0738_Temp.png
210514_0738_Temp.png
Attachment 10: 210514_0739_Bias.png
210514_0739_Bias.png
Attachment 11: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi.png
Screenshot_2021-05-14_Statistics_aidas-gsi.png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Attachment 18: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Attachment 19: Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Attachment 20: 600.png
600.png
Attachment 21: 601.png
601.png
  314   Thu May 13 21:21:25 2021 Philippos and Marcsystem wide checks

Wide checks at Time 22:22 CET

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dbc :      4  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

             Base         Current         Difference
aida05 fault      0x0 :      0x2 :      2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     35     11      3      5      5      4      1      3      2      4      6   : 37876
aida02 :     20      5      4      3      2      5      2      4      3      3      6   : 37400
aida03 :     20      3      4      3      4      4      1      3      2      4      6   : 37640
aida04 :     25      6      2      3      4      4      3      3      3      3      6   : 37140
aida05 :     30     10      5      7      2      6      2      4      2      4      6   : 38776
aida06 :     21     11      4      4      3      5      2      4      3      3      6   : 37548
aida07 :     23      8      4      4      2      6      1      3      2      4      6   : 37852
aida08 :     27      7      2      3      4      3      2      4      3      3      6   : 37284
aida09 :     31      6      5      6      3      3      2      4      3      3      6   : 37372
aida10 :     22      5      3      4      3      6      2      2      2      4      6   : 37616
aida11 :     23      5      4      2      5      2      1      3      2      4      6   : 37444
aida12 :     20      7      4      7      3      5      2      3      3      3      6   : 37096
aida13 :     28      5      3      3      1      5      2      3      3      3      6   : 36840
aida14 :     38     12      2      4      2      4      1      4      3      3      6   : 37144
aida15 :     27      9      5      4      2      5      3      2      3      3      6   : 36740
aida16 :     21      6      4      4      1      4      2      3      3      3      6   : 36740

 

Attachment 1: 2021-05-13_ucesb_scalers_22h22mn.png
2021-05-13_ucesb_scalers_22h22mn.png
Attachment 2: 2021-05-13_Currents_22h22mn.png
2021-05-13_Currents_22h22mn.png
Attachment 3: 2021-05-13_temperatures_22h22.png
2021-05-13_temperatures_22h22.png
Attachment 4: 2021-05-13-22h22mn_Stat-good-events.png
2021-05-13-22h22mn_Stat-good-events.png
Attachment 5: 2021-05-13-22h22mn_ADC-data-items.png
2021-05-13-22h22mn_ADC-data-items.png
Attachment 6: 2021-05-13_22h22mn-Correlation-info.png
2021-05-13_22h22mn-Correlation-info.png
Attachment 7: 2021-05-13_22h22mn-Pause-info.png
2021-05-13_22h22mn-Pause-info.png
Attachment 8: 2021-05-13_22h22mn-Resume-Info.png
2021-05-13_22h22mn-Resume-Info.png
Attachment 9: 2021-05-13_merger_22h22.png
2021-05-13_merger_22h22.png
Attachment 10: 2021-05-13_22h22mn-LAY-ID1.png
2021-05-13_22h22mn-LAY-ID1.png
Attachment 11: 2021-05-13_22h22mn-LAY-ID2.png
2021-05-13_22h22mn-LAY-ID2.png
Attachment 12: 2021-05-13_22h22mn-LAY-ID3.png
2021-05-13_22h22mn-LAY-ID3.png
Attachment 13: 2021-05-13_22h22mn-LAY-ID4.png
2021-05-13_22h22mn-LAY-ID4.png
  313   Thu May 13 19:52:36 2021 Philippos and MarcConnection problem

At 20:15, We lost connection with AnyDesk.

Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter.

Helena restarted all after ~30mn and it's all working again now.

 

 

Attachment 1: 2021-05-13_ucesb_scalers.png
2021-05-13_ucesb_scalers.png
  312   Thu May 13 17:05:42 2021 Muneerah, MArcThu 13 May 04:00-00:00

19:08 System Check

Rates from ucesb below 1 kHz

Atachment 1 Stats

Atachment 2 Temperatures

Atachment 3 Voltages

System wide checks are all ok except

ADC calibration all failed  
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White rabbit  
         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dba :      2  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA all passed

Memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6      2      2      1      2      4      3      4      2      3      7   : 40424
aida02 :      3      4      2      2      1      2      1      3      3      3      7   : 40140
aida03 :     10     10      0      3      2      4      2      4      2      3      7   : 40280
aida04 :      3      2      0      2      2      3      2      3      1      4      7   : 40540
aida05 :      4      9      2      3      2      4      3      3      1      4      7   : 41048
aida06 :      8      6      1      1      2      3      2      2      3      3      7   : 40064
aida07 :      1      4      0      1      1      3      1      3      3      3      7   : 40196
aida08 :      1      4      2      2      2      4      3      4      2      3      7   : 40452
aida09 :      8      5      2      2      1      2      1      3      3      3      7   : 40168
aida10 :     22      5      4      2      1      3      1      4      2      3      7   : 39872
aida11 :      2      1      2      1      2      2      3      3      2      3      7   : 39632
aida12 :      8      5      2      2      1      3      1      4      2      3      7   : 39784
aida13 :     18      8      2      1      0      2      3      3      2      3      7   : 39624
aida14 :     23     12      2      2      1      2      2      4      2      3      7   : 40028
aida15 :      3      5      3      2      2      3      1      4      2      3      7   : 39844
aida16 :      0      2      1      2      1      4      2      4      2      3      7   : 40096

 

 

 

Attachment 1: 130521_1802_Stats.png
130521_1802_Stats.png
Attachment 2: 130521_1802_Temp.png
130521_1802_Temp.png
Attachment 3: 130521_1804_Voltages.png
130521_1804_Voltages.png
  311   Thu May 13 07:10:17 2021 CA, TDMay 13th 08:00 - 16:00 shift
08:10 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

08:16 attempt to recalibrate ADCs using FADC align and control - unsuccessful on all

08:18 FEE64 temps ok - attachment 1

      bias/leakage currents ok - attachment 2

      statistics - attachments 3-8

10:13 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

10:16 FEE64 temps ok - attachment 9

      bias/leakage currents ok - attachment 10

      statistics - attachments 11-16

12:16 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

12:16 FEE64 temps ok - attachment 17

      bias/leakage currents ok - attachment 18

      statistics - attachments 19-25

14:08 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FEE64 temps ok - attachment 16

      bias/leakage currents ok - attachment 27

      statistics - attachments 28-35
Attachment 1: Screenshot_from_2021-05-13_07-17-47.png
Screenshot_from_2021-05-13_07-17-47.png
Attachment 2: Screenshot_from_2021-05-13_07-18-57.png
Screenshot_from_2021-05-13_07-18-57.png
Attachment 3: Screenshot_from_2021-05-13_07-19-42.png
Screenshot_from_2021-05-13_07-19-42.png
Attachment 4: Screenshot_from_2021-05-13_07-20-45.png
Screenshot_from_2021-05-13_07-20-45.png
Attachment 5: Screenshot_from_2021-05-13_07-23-01.png
Screenshot_from_2021-05-13_07-23-01.png
Attachment 6: Screenshot_from_2021-05-13_07-23-35.png
Screenshot_from_2021-05-13_07-23-35.png
Attachment 7: Screenshot_from_2021-05-13_07-24-02.png
Screenshot_from_2021-05-13_07-24-02.png
Attachment 8: Screenshot_from_2021-05-13_07-25-15.png
Screenshot_from_2021-05-13_07-25-15.png
Attachment 9: Screenshot_from_2021-05-13_09-17-57.png
Screenshot_from_2021-05-13_09-17-57.png
Attachment 10: Screenshot_from_2021-05-13_09-18-21.png
Screenshot_from_2021-05-13_09-18-21.png
Attachment 11: Screenshot_from_2021-05-13_09-18-58.png
Screenshot_from_2021-05-13_09-18-58.png
Attachment 12: Screenshot_from_2021-05-13_09-19-32.png
Screenshot_from_2021-05-13_09-19-32.png
Attachment 13: Screenshot_from_2021-05-13_09-20-23.png
Screenshot_from_2021-05-13_09-20-23.png
Attachment 14: Screenshot_from_2021-05-13_09-21-05.png
Screenshot_from_2021-05-13_09-21-05.png
Attachment 15: Screenshot_from_2021-05-13_09-21-53.png
Screenshot_from_2021-05-13_09-21-53.png
Attachment 16: Screenshot_from_2021-05-13_09-22-18.png
Screenshot_from_2021-05-13_09-22-18.png
Attachment 17: Screenshot_from_2021-05-13_11-19-16.png
Screenshot_from_2021-05-13_11-19-16.png
Attachment 18: Screenshot_from_2021-05-13_11-19-30.png
Screenshot_from_2021-05-13_11-19-30.png
Attachment 19: Screenshot_from_2021-05-13_11-20-09.png
Screenshot_from_2021-05-13_11-20-09.png
Attachment 20: Screenshot_from_2021-05-13_11-20-33.png
Screenshot_from_2021-05-13_11-20-33.png
Attachment 21: Screenshot_from_2021-05-13_11-20-59.png
Screenshot_from_2021-05-13_11-20-59.png
Attachment 22: Screenshot_from_2021-05-13_11-21-29.png
Screenshot_from_2021-05-13_11-21-29.png
Attachment 23: Screenshot_from_2021-05-13_11-21-53.png
Screenshot_from_2021-05-13_11-21-53.png
Attachment 24: Screenshot_from_2021-05-13_11-22-30.png
Screenshot_from_2021-05-13_11-22-30.png
Attachment 25: Screenshot_from_2021-05-13_11-24-36.png
Screenshot_from_2021-05-13_11-24-36.png
Attachment 26: Screenshot_from_2021-05-13_13-09-43.png
Screenshot_from_2021-05-13_13-09-43.png
Attachment 27: Screenshot_from_2021-05-13_13-10-03.png
Screenshot_from_2021-05-13_13-10-03.png
Attachment 28: Screenshot_from_2021-05-13_13-10-28.png
Screenshot_from_2021-05-13_13-10-28.png
Attachment 29: Screenshot_from_2021-05-13_13-10-52.png
Screenshot_from_2021-05-13_13-10-52.png
Attachment 30: Screenshot_from_2021-05-13_13-11-22.png
Screenshot_from_2021-05-13_13-11-22.png
Attachment 31: Screenshot_from_2021-05-13_13-11-51.png
Screenshot_from_2021-05-13_13-11-51.png
Attachment 32: Screenshot_from_2021-05-13_13-12-24.png
Screenshot_from_2021-05-13_13-12-24.png
Attachment 33: Screenshot_from_2021-05-13_13-13-34.png
Screenshot_from_2021-05-13_13-13-34.png
Attachment 34: Screenshot_from_2021-05-13_13-14-16.png
Screenshot_from_2021-05-13_13-14-16.png
  310   Thu May 13 00:51:40 2021 Muneerah and OHThu 13 May 00:00-08:00
01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK

Rates OK

Attachment 1 Voltages

Atachment 2 Rates

Atachment 3 Temperatures

Attachment 4 ucesb rates less than 1 kHz


System wide check are ok except:

Base Current Difference
aida05 fault  0x36ca :  0x36ce :  4 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base Current Difference
aida13 fault 0xa :  0xe9 :  223 
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716



<strong>05:00 OH Took over at 04:00</strong>

System wide checks ok except

   Base   Current  Difference
aida05 fault   0x36ca :   0x36d0 :   6 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


    Base   Current   Difference
aida05 fault   0x0 :   0x1 :   1 
aida13 fault   0xa :   0xe9 :   223 
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



05:41 AIDA 05 has dropped out. Will restart the system

05:58 DAQ recovered from powercycle
 p+n strips at 0xa
 n+n strips at 0x20
 ADC control register at 0xFF
 All waveforms turned off in LED

 Statistics - attachment 5
 Correlation scaler in FEE 3 and 4 are mostly tracking 1:1

06:34 Local users have been unable to recover MBS
      And have been unable to contact local experts

07:22 Bias and leakage currents ok - attachment 6
 Stats attachment -7
Attachment 1: 130521_0155_Voltages.png
130521_0155_Voltages.png
Attachment 2: 130521_0157_Stats.png
130521_0157_Stats.png
Attachment 3: 130521_0158_Temp.png
130521_0158_Temp.png
Attachment 4: 130521_0204_ucesb.png
130521_0204_ucesb.png
Attachment 5: 210513_0600_Stats.png
210513_0600_Stats.png
Attachment 6: 210513_0722_bias.png
210513_0722_bias.png
Attachment 7: 210513_0722_Stats.png
210513_0722_Stats.png
  309   Wed May 12 18:53:38 2021 JSWed 12 20:00-00:00
19:51 Shift change

Stats ok
ucesb ok, Max implants ~1500 Hz 

20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)

20:51 
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.

20:59 

Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36ce : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0xb6 : 	 172  
FPGA Timestamp error counter test result: Passed 15, Failed 1

--- both errors above same module as earlier shift
	 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      3      2      0      1      3      3      2      3      3      6   : 36092
aida02 :      6      4      4      2      2      3      1      3      3      3      6   : 36280
aida03 :      6      3      2      0      0      3      3      2      3      3      6   : 36048
aida04 :     10      3      1      1      3      3      1      3      3      3      6   : 36272
aida05 :     19      8      2      0      2      3      1      3      3      3      6   : 36268
aida06 :      2      2      0      2      3      4      2      2      3      3      6   : 36120
aida07 :      6      7      3      0      3      4      3      2      3      3      6   : 36416
aida08 :      7      5      4      0      3      2      2      3      3      3      6   : 36420
aida09 :     21      6      3      2      2      4      2      3      2      3      6   : 35700
aida10 :     13      3      1      1      3      3      1      4      2      3      6   : 35772
aida11 :     25      6      4      2      1      4      2      4      2      3      6   : 36180
aida12 :      0      4      0      3      2      3      3      3      2      3      6   : 35712
aida13 :     18      5      2      3      2      3      2      4      2      3      6   : 36080
aida14 :      2      1      1      1      2      3      2      4      2      3      6   : 35904
aida15 :      0      1      1      3      0      4      2      2      3      3      6   : 35960
aida16 :      4      5      3      2      3      4      1      3      2      3      6   : 35432

ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4

21:36
ucesb ok
Stats ok 

22:01
ucesb ok 1500 Hz peak implant
Stats ok 
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.

22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler

23:40 FRS magnet issue, possibly power, no beam

00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah
Attachment 1: ucesb.png
ucesb.png
Attachment 2: bias1.png
bias1.png
Attachment 3: Screenshot_2021-05-12_Statistics_aidas-gsi.png
Screenshot_2021-05-12_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: bias2.png
bias2.png
  308   Wed May 12 15:03:10 2021 CB, OHWed 12 16:00-20:00
16:00 Shift change

Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)

System wide checks
Clock, ADC - OK

WR
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36cb : 	 1  
White Rabbit error counter test result: Passed 15, Failed 1


FPGA

			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0x4b : 	 65  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      5      6      1      1      2      2      3      3      3      6   : 36388
aida02 :     21     10      2      2      2      3      2      2      3      3      6   : 36100
aida03 :     25      5      3      1      0      3      3      2      3      3      6   : 36188
aida04 :      5      2      0      3      2      3      2      3      3      3      6   : 36484
aida05 :     16      6      3      1      2      4      1      3      2      4      6   : 37440
aida06 :      2      2      2      2      2      3      1      3      3      3      6   : 36216
aida07 :     16      9      3      0      3      4      3      2      3      3      6   : 36472
aida08 :      0      7      1      1      2      2      3      3      2      3      6   : 35560
aida09 :     15      7      4      2      2      4      2      3      2      3      6   : 35700
aida10 :      3      7      3      4      1      3      1      4      1      4      6   : 36788
aida11 :      5      1      3      1      1      4      3      3      2      3      6   : 35756
aida12 :      5      5      0      3      2      3      2      4      2      3      6   : 35996
aida13 :      2      5      1      2      2      4      3      3      2      3      6   : 35840
aida14 :     19      5      3      1      1      2      1      3      3      3      6   : 36100
aida15 :     20      6      6      2      1      3      2      2      3      3      6   : 36064
aida16 :      7      9      4      1      3      4      2      3      2      3      6   : 35716

File size change
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 10 16:47:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.

17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before

17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK

17:52 Started compressing R1_1*

18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before

19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)

19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before

19:51 Shift handed over to James S
Attachment 1: Stat1.png
Stat1.png
Attachment 2: Temp1.png
Temp1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Stats2.png
Stats2.png
Attachment 5: Bias2.png
Bias2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: ucesb1.png
ucesb1.png
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Bias4.png
Bias4.png
Attachment 11: Merger.png
Merger.png
Attachment 12: ucesb2.png
ucesb2.png
Attachment 13: Stats5.png
Stats5.png
Attachment 14: Temps2.png
Temps2.png
Attachment 15: Bias5.png
Bias5.png
  307   Wed May 12 06:59:55 2021 OH, RDPWed 12 May 08:00-16:00
08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...
Attachment 1: 1dImplants.png
1dImplants.png
Attachment 2: 1dImplants2.png
1dImplants2.png
Attachment 3: 512.png
512.png
Attachment 4: 513.png
513.png
  306   Wed May 12 02:07:59 2021 MA, BAWed 12 May 00:00-08:00

03:12 System Check

attachment 1 : Current

attachment 2 : Temperature

attachment 3 : Rates

 

   
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

   
Calibration test result: Passed 16, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida13 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      5      7      2      1      2      2      3      3      3      6   : 36464
aida02 :      9      8      3      3      1      4      1      2      4      3      6   : 36916
aida03 :      5      2      6      2      0      3      2      3      3      3      6   : 36420
aida04 :      6      5      2      3      4      4      2      3      3      3      6   : 36800
aida05 :     17      6      6      2      2      4      1      3      2      4      6   : 37524
aida06 :      7     12      3      4      3      3      1      3      3      3      6   : 36460
aida07 :     17     11      5      1      3      3      3      2      3      3      6   : 36428
aida08 :      3      5      0      1      4      2      2      4      2      3      6   : 35924
aida09 :     27      6      4      2      0      2      2      2      3      3      6   : 35868
aida10 :     16     11      8      0      2      2      1      3      2      4      6   : 37272
aida11 :     15      2      2      3      1      4      3      4      2      3      6   : 36364
aida12 :      1      6      4      3      1      3      2      4      2      3      6   : 35988
aida13 :     22     14     10      2      4      2      2      4      2      3      6   : 36264
aida14 :     26     10      4      3      2      1      1      3      3      3      6   : 36184
aida15 :     14      2      3      2      2      4      1      2      3      3      6   : 35896
aida16 :      7      5      4      0      2      4      2      3      2      3      6   : 35588

06:10 DSSD1 rate high!

Attached 4

Called OH and he woke up to fix it :)

It was a problem with one of the ASIC according to what he says:

One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35

 

07:03 System check

attachment 5 Spectrum rate

attachment 6 Voltages

attachment 7 Rates

attachment 8 Temperature

System wide clock are all ok except

White rabbit

 Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA

 Base         Current         Difference
aida13 fault      0xa :      0x14 :      10  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: 1205_0310.png
1205_0310.png
Attachment 2: 1205_0311.png
1205_0311.png
Attachment 3: 1205_0312.png
1205_0312.png
Attachment 4: 120521_0630ucesb.png
120521_0630ucesb.png
Attachment 5: 120521_0639_SpecRates.png
120521_0639_SpecRates.png
Attachment 6: 120521_0655_Voltages.png
120521_0655_Voltages.png
Attachment 7: 120521_0657_Rates.png
120521_0657_Rates.png
Attachment 8: 120521_0685_Temp.png
120521_0685_Temp.png
Attachment 9: 120521_0635_ASICs.png
120521_0635_ASICs.png
  305   Tue May 11 07:21:24 2021 OHTuesday 11 May
08:20 System wide checks all ok except
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

and

	
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    Statistics still high - attachment 1
    Temperatures ok - attachment 2
    Bias and leakage currents ok - attachment 3
Attachment 1: 210511_0823_Stats.png
210511_0823_Stats.png
Attachment 2: 210511_0824_Temp.png
210511_0824_Temp.png
Attachment 3: 210511_0825_Bias.png
210511_0825_Bias.png
  304   Mon May 10 11:45:15 2021 NHConfusing behaviour of rates
Testing the behaviour of powering down the fast ADCs in aida01 (the hot FEE)

Set local control 2 to 0xff in aida01
Wait... see temperature drpos by ~3 degrees (PJCS reports much larger drop in T9)
So far so normal

*but*

aida05 rates go very high when this waveform is disabled!!
Set local control to 0x00 again
They drop back...

Recalibrated FADCs and back to normal
  303   Mon May 10 08:49:37 2021 TDMonday 10 May
09.51 All system wide checks OK *except*


		 Base 		Current 	Difference
aida05 fault 	 0x8441 : 	 0x8442 : 	 1  
aida06 fault 	 0xc5c5 : 	 0xc5c6 : 	 1  
aida07 fault 	 0x6f07 : 	 0x6f08 : 	 1  
aida08 fault 	 0x4814 : 	 0x4815 : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      6      3      5      5      2      3      2      3      6   : 36004
aida02 :      4      1      3      6      3      4      3      3      2      3      6   : 36040
aida03 :     12      8      5      2      3      3      3      4      2      3      6   : 36416
aida04 :      1      2      5      4      1      4      3      4      2      3      6   : 36388
aida05 :     22      8      4      2      4      5      1      4      1      4      6   : 37272
aida06 :      5      4      0      6      3      6      3      3      2      3      6   : 36276
aida07 :     38     16      7      7      1      6      1      3      3      3      6   : 37032
aida08 :     16      4      2      3      1      4      1      3      3      3      6   : 36384
aida09 :      2      3      7      6      5      5      2      3      2      3      6   : 36112
aida10 :     42     15      4      6      2      6      2      4      2      3      6   : 36768
aida11 :      1      5     13      7      3      3      3      3      2      3      6   : 36124
aida12 :     23      9      3      4      7      3      2      3      2      3      6   : 35988
aida13 :      2      6      7      3      4      4      3      2      2      3      6   : 35592
aida14 :     17     11      3      4      3      5      2      3      2      3      6   : 35980
aida15 :      4      7      5      2      1      6      2      3      2      3      6   : 35864
aida16 :      4      1      3      3      4      2      3      4      2      3      6   : 36264


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 09 15:10:52 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.52 All histograms, stats and merger stats zero'd

      FEE64 temps OK - attachment 1
      ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6
      Rate spectra - attachment 7
       ASIC check load
      Rate spectra - attachment 8
      1.8.W spectra 20us FSR - attachments 9 & 10
      Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11
      Merger stats/tape server/merger - attachment 12
      DSSSD bias and leakage current OK - attachment 13

10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21
       we are not missing extrinsic noise features at longer time ranges cf. 20us FSR

12.18 Cannot connect to port 8015, MIDAS terminal says "Killed"
      Restart MIDAS 8015 
      Collect FPGA baseline to avoid TCL error
      FEEs unaffected and running as before

13.25 FEEs power cycled to see if any change in weird noise/rates situation
      No change 
      ASICs synchronised and back to normal

14.34 Start a pulser walkthrough in May21/R10
      Attenuation 5x, Rep Rate 25 Hz
      Amplitudes 10..9...1

14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again.

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0 (NB. actually it is 12 MeV)
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
Attachment 1: Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-10_Statistics_aidas-gsi.png
Screenshot_2021-05-10_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Attachment 7: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Attachment 12: 510.png
510.png
Attachment 13: 511.png
511.png
Attachment 14: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 15: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Attachment 16: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 17: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Attachment 18: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 19: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Attachment 20: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 21: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
  302   Sun May 9 19:29:23 2021 TDSunday 9 May
Attachment 1: Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-09_Statistics_aidas-gsi.png
Screenshot_2021-05-09_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Attachment 7: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Attachment 12: Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Attachment 13: 503.png
503.png
Attachment 14: 504.png
504.png
  301   Sat May 8 09:05:04 2021 TDSaturday 8 May
09.54 FEE64 temperatures OK - attachment 1
      ADC data item stats OK - attachment 2
       p+n FEE64s 11x <100k, 1x <200k
      Rate spectra - attachments 3 & 4
       before and after ASIC check load
      1.8.L spectra - attachments 5 & 6
      1.8.W spectra 20us FSR - attachments 7 & 8
      Disc, WR-28-47, pause, resume info data items - attachments 9-12
       note large number pause/resume for aida13 - legacy numbers since last restart/zero?
       attachment 16 for pause stats since stats zero at 10.02 - looks OK
      Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13
      DSSSD bias & leakage currents - attachment 14
      Merger stats/merger/tape server messages - attachment 15

10.02 All histograms and stats zero'd
      
10.06 All system wide checks OK *except*

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 06 15:29:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM
       peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298
Attachment 1: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-08_Statistics_aidas-gsi.png
Screenshot_2021-05-08_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png
Attachment 6: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png
Attachment 7: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png
Attachment 8: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png
Attachment 9: Screenshot_2021-05-08_Statistics_aidas-gsi(1).png
Screenshot_2021-05-08_Statistics_aidas-gsi(1).png
Attachment 10: Screenshot_2021-05-08_Statistics_aidas-gsi(2).png
Screenshot_2021-05-08_Statistics_aidas-gsi(2).png
Attachment 11: Screenshot_2021-05-08_Statistics_aidas-gsi(3).png
Screenshot_2021-05-08_Statistics_aidas-gsi(3).png
Attachment 12: Screenshot_2021-05-08_Statistics_aidas-gsi(4).png
Screenshot_2021-05-08_Statistics_aidas-gsi(4).png
Attachment 13: Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png
Attachment 14: 500.png
500.png
Attachment 15: 501.png
501.png
Attachment 16: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Attachment 17: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Attachment 18: Screenshot_2021-05-08_Statistics_aidas-gsi(6).png
Screenshot_2021-05-08_Statistics_aidas-gsi(6).png
Attachment 19: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 20: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png
Attachment 21: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png
Attachment 22: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png
Attachment 23: Screenshot_2021-05-08_Statistics_aidas-gsi(7).png
Screenshot_2021-05-08_Statistics_aidas-gsi(7).png
Attachment 24: Screenshot_2021-05-08_Statistics_aidas-gsi(8).png
Screenshot_2021-05-08_Statistics_aidas-gsi(8).png
Attachment 25: Screenshot_2021-05-08_Statistics_aidas-gsi(9).png
Screenshot_2021-05-08_Statistics_aidas-gsi(9).png
Attachment 26: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png
Attachment 27: Screenshot_2021-05-08_Statistics_aidas-gsi(10).png
Screenshot_2021-05-08_Statistics_aidas-gsi(10).png
  300   Fri May 7 19:47:54 2021 PJCSHowTo mitigate excessive temperature in an FEE64

After tests in the Daresbury T9 system.

Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees.

To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF

The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected.

Alternatively STOP acquisition, set the ADC Control  register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.

 

  299   Fri May 7 13:11:22 2021 NHFriday 7th May
14:11 - Alpha has been running most of morning

Just saw rates in tape spike to 6 MB/s... 
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s

R6_49 will be affected by this.
Others seem OK

System wide check failures:

		 Base 		Current 	Difference
aida01 fault 	 0xb405 : 	 0xb406 : 	 1  
aida02 fault 	 0xefc7 : 	 0xefc8 : 	 1  
aida03 fault 	 0xdaab : 	 0xdaac : 	 1  
aida04 fault 	 0x8f7c : 	 0x8f7d : 	 1  
aida05 fault 	 0xb5bd : 	 0xb5be : 	 1  
aida06 fault 	 0xeff1 : 	 0xeff2 : 	 1  
aida07 fault 	 0x8f57 : 	 0x8f58 : 	 1  
aida08 fault 	 0xbef7 : 	 0xbef8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x200 : 	 512  
aida13 fault 	 0x0 : 	 0x61 : 	 97  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)

Temps & Rates OK

Will now stop Alpha run and check noise situation

Run stopped at R6_49

Switching to nostorage (NOTAPE/R2)

Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s

No change to situation since yesterday... good sign

During MBS test aida13 rebooted... crash log from ttyUSB5 attached

It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)

A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK
Attachment 1: elog_locks.png
elog_locks.png
Attachment 2: elog_temps.png
elog_temps.png
Attachment 3: elog_rates.png
elog_rates.png
Attachment 4: elog_rates1.png
elog_rates1.png
Attachment 5: aida13.txt
06/19:25:39|Data Acquisition Statistics counters now cleared^M
06/19:25:55|Clear Statistics (1)^M
06/19:25:55|------------[ cut here ]------------^M
07/15:23:58|kernel BUG at mm/slab.c:2974!^M
07/15:23:58|Oops: Exception in kernel mode, sig: 5 [#1]^M
07/15:23:58|PREEMPT Xilinx Virtex440^M
07/15:23:59|Modules linked in: aidamem xdriver xh_spidev_register^M
07/15:23:59|NIP: c009211c LR: c00920b0 CTR: 00000007^M
07/15:23:59|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)^M
07/15:23:59|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000^M
07/15:23:59|TASK = c036e318[0] 'swapper' THREAD: c0390000^M
07/15:23:59|GPR00: 00000001 c0391ca0 c036e318 c680daf0 c694003c 0000000a c6940020 00000009 ^M
07/15:23:59|GPR08: 0000001b c680dae0 00000cf0 c680dae0 24008042 00005aa8 c03a0000 00000020 ^M
07/15:23:59|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 ^M
07/15:23:59|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000006 c680e400 ^M
07/15:23:59|NIP [c009211c] cache_alloc_refill+0x130/0x608^M
07/15:23:59|LR [c00920b0] cache_alloc_refill+0xc4/0x608^M
07/15:23:59|Call Trace:^M
07/15:23:59|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)^M
07/15:23:59|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M
07/15:23:59|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M
07/15:23:59|[c0391d40] [c0042938] __send_signal+0x78/0x260^M
07/15:23:59|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M
07/15:24:00|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M
07/15:24:00|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M
07/15:24:00|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M
07/15:24:00|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M
07/15:24:00|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M
07/15:24:00|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M
07/15:24:00|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M
07/15:24:00|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M
07/15:24:00|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M
07/15:24:00|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M
07/15:24:00|[c0391fb0] [c000172c] rest_init+0x70/0x84^M
07/15:24:00|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M
07/15:24:00|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M
07/15:24:00|Instruction dump:^M
07/15:24:00|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 ^M
07/15:24:00|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 ^M
07/15:24:00|Kernel panic - not syncing: Fatal exception in interrupt^M
07/15:24:00|Call Trace:^M
07/15:24:00|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)^M
07/15:24:00|[c0391a80] [c00345bc] panic+0x94/0x168^M
07/15:24:01|[c0391ad0] [c000bd44] die+0x178/0x18c^M
07/15:24:01|[c0391af0] [c000c000] _exception+0x164/0x1b4^M
07/15:24:01|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c^M
07/15:24:01|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608^M
07/15:24:01|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M
07/15:24:01|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M
07/15:24:01|[c0391d40] [c0042938] __send_signal+0x78/0x260^M
07/15:24:01|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M
07/15:24:01|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M
07/15:24:01|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M
07/15:24:01|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M
07/15:24:01|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M
07/15:24:01|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M
07/15:24:01|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M
07/15:24:01|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M
07/15:24:01|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M
07/15:24:01|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M
07/15:24:01|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M
07/15:24:01|[c0391fb0] [c000172c] rest_init+0x70/0x84^M
07/15:24:01|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M
07/15:24:02|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M
07/15:24:02|Rebooting in 180 seconds..
07/15:27:02|^MISOL Version 1.00 Date 9th January 2017
  298   Thu May 6 16:59:20 2021 TD, OHWednesday 6 May
18.09 NH has installed jumpers LK2-4 for *all* adaptor PCBs

      per FEE64 rate spectra - attachments 1-4
       attachment 4 with BNC PB-5 pulser 10Hz
      ADC data items stats - attachment 5
      1.8.W spectra - attachments 6 & 7

      Observe significant reduction in DSSSD/ribbon cable boundary channel rates
       - status quo ante Tuesday 4 May https://elog.ph.ed.ac.uk/DESPEC/291 ?


      1.8.L spectra - attachments 8 & 9
       pulser peak width aida01 85 ch FWHM, aida2 295 ch FWHM

19:21 Alpha run started May21/R6
      Statistics dominated by aida04 with a rate of 6k
      System wide checks all ok except:
			 Base 		Current 		Difference
aida13 fault 	 0x0 : 	 0x17 : 	 23  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Writing to file at ~300kB/s
     Rate spectra for aida04 shows dominated by first asic - attachment 10
     1*L spectra for aida04 asic 1 - attachment 11

20.25 per FEE64 rate spectra - attachment 12
      handful of hot channels aida06, 09 & 10
      ADC data item stats - attachment 13
Attachment 1: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png
Attachment 2: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png
Attachment 3: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png
Attachment 4: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png
Attachment 5: Screenshot_2021-05-06_Statistics_aidas-gsi(1).png
Screenshot_2021-05-06_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png
Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png
Attachment 8: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Attachment 9: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 10: 210506_1924_aida04Rate.png
210506_1924_aida04Rate.png
Attachment 11: 210506_1927_1*L.png
210506_1927_1*L.png
Attachment 12: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png
Attachment 13: Screenshot_2021-05-06_Statistics_aidas-gsi.png
Screenshot_2021-05-06_Statistics_aidas-gsi.png
  297   Wed May 5 20:10:16 2021 OHWednesday 5 May Alpha run
21:10 After all of the works in S4 today the noise in the system was considerably worse
      It was decided that before starting a new alpha run a powercycle would be performed.
      The powercycle has had no effect on the rates. They are still terrible.

      The difference between yesterday and today is:
      - Adaptor cards for FEE9 and 13 were re-aligned as they were previously misaligned
      - bPlast has been plugged in and biased

      All system wide checks ok
      Statistics threshold at 0xa - attachment 1
      Rate spectra (FEE5 does have events in all channels just noise is dominated by a few) - attachment 2
      Waveforms - attachments 3 and 4

      Pulser turned off and threshold raised to 0x64
      Statistics - attachment 5

21:19 Alpha run started to May21/R4*


08:51 DAQ continues OK - file May21/R4_23

      FEE64 temps OK - attachment 8
      ADC data items OK - attachment 9 
      1.8.W spectra - attachments 6 & 7
       no change cf. yesterday pm
      Grafana - most recent 12h DSSSD bias & leakage current - attachment 10
      DSSSD biases & leakage currents - attachment 11
      Tape Server/Merger - attachment 12

 09:44 R4 stopped
Attachment 1: 210504_2105_Stats.png
210504_2105_Stats.png
Attachment 2: 210504_2107_Layout1.png
210504_2107_Layout1.png
Attachment 3: 210504_2108_layout7.png
210504_2108_layout7.png
Attachment 4: 210504_2109_Layout8.png
210504_2109_Layout8.png
Attachment 5: 210504_2115_stats.png
210504_2115_stats.png
Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-05-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_2021-05-06_Statistics_aidas-gsi.png
Screenshot_2021-05-06_Statistics_aidas-gsi.png
Attachment 10: Screenshot_2021-05-06_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-06_AIDA_Alerting_-_Grafana.png
Attachment 11: 401.png
401.png
Attachment 12: 402.png
402.png
  296   Wed May 5 16:57:48 2021 TDTuesday 5 May
17.55 Grafana DSSSD bias & leakage current - attachment 1
       Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37 
      DSSSD bias & leakage currents - attachment 2
Attachment 1: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-05-05_AIDA_Alerting_-_Grafana(1).png
Attachment 2: 401.png
401.png
  295   Tue May 4 21:26:43 2021 OHAlpha run
22:26 Tape server set up to write to directory May21
      DAQ stopped
      Slow comparator set to 0x64 for all FEE64 and an ASIC check performed
      Pulser turned off
      DAQ started
      ~300k in merger - 38kB per second to file
      Statistics good (all FEE show 0 bar FEE4 at 3k) - attachment 1
      All system wide checks ok apart from FPGA check
      	
	        Base	Current  	Difference
aida13 fault 	 0x0 : 	 0x62 : 	 98  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Histograms zeroed.
     DSSD1 at 130V
     DSSD2 at 100V

08:47 Found DAQ running at c. 3M data items/s / 13MB/s
      Switched Tape Server to no storage mode
      
[npg@aidas-gsi ~]$ ls -lrth /TapeData/May21
total 113G
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_0
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_1
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_2
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_3
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_4
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_5
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_6
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_7
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_8
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_9
-rw-rw-r--. 1 npg npg 1.3G May  5 06:15 R1_10
-rw-rw-r--. 1 npg npg 2.0G May  5 06:17 R1_11
-rw-rw-r--. 1 npg npg 2.0G May  5 06:20 R1_12
-rw-rw-r--. 1 npg npg 2.0G May  5 06:23 R1_13
-rw-rw-r--. 1 npg npg 2.0G May  5 06:25 R1_14
-rw-rw-r--. 1 npg npg 2.0G May  5 06:28 R1_15
-rw-rw-r--. 1 npg npg 2.0G May  5 06:30 R1_16
-rw-rw-r--. 1 npg npg 2.0G May  5 06:33 R1_17
-rw-rw-r--. 1 npg npg 2.0G May  5 06:36 R1_18
-rw-rw-r--. 1 npg npg 2.0G May  5 06:38 R1_19
-rw-rw-r--. 1 npg npg 2.0G May  5 06:41 R1_20
-rw-rw-r--. 1 npg npg 2.0G May  5 06:43 R1_21
-rw-rw-r--. 1 npg npg 2.0G May  5 06:46 R1_22
-rw-rw-r--. 1 npg npg 2.0G May  5 06:49 R1_23
-rw-rw-r--. 1 npg npg 2.0G May  5 06:51 R1_24
-rw-rw-r--. 1 npg npg 2.0G May  5 06:54 R1_25
-rw-rw-r--. 1 npg npg 2.0G May  5 06:57 R1_26
-rw-rw-r--. 1 npg npg 2.0G May  5 06:59 R1_27
-rw-rw-r--. 1 npg npg 2.0G May  5 07:02 R1_28
-rw-rw-r--. 1 npg npg 2.0G May  5 07:04 R1_29
-rw-rw-r--. 1 npg npg 2.0G May  5 07:07 R1_30
-rw-rw-r--. 1 npg npg 2.0G May  5 07:10 R1_31
-rw-rw-r--. 1 npg npg 2.0G May  5 07:12 R1_32
-rw-rw-r--. 1 npg npg 2.0G May  5 07:15 R1_33
-rw-rw-r--. 1 npg npg 2.0G May  5 07:18 R1_34
-rw-rw-r--. 1 npg npg 2.0G May  5 07:20 R1_35
-rw-rw-r--. 1 npg npg 2.0G May  5 07:23 R1_36
-rw-rw-r--. 1 npg npg 2.0G May  5 07:25 R1_37
-rw-rw-r--. 1 npg npg 2.0G May  5 07:28 R1_38
-rw-rw-r--. 1 npg npg 2.0G May  5 07:31 R1_39
-rw-rw-r--. 1 npg npg 2.0G May  5 07:33 R1_40
-rw-rw-r--. 1 npg npg 2.0G May  5 07:36 R1_41
-rw-rw-r--. 1 npg npg 2.0G May  5 07:39 R1_42
-rw-rw-r--. 1 npg npg 2.0G May  5 07:41 R1_43
-rw-rw-r--. 1 npg npg 2.0G May  5 07:44 R1_44
-rw-rw-r--. 1 npg npg 2.0G May  5 07:46 R1_45
-rw-rw-r--. 1 npg npg 2.0G May  5 07:49 R1_46
-rw-rw-r--. 1 npg npg 2.0G May  5 07:52 R1_47
-rw-rw-r--. 1 npg npg 2.0G May  5 07:54 R1_48
-rw-rw-r--. 1 npg npg 2.0G May  5 07:57 R1_49
-rw-rw-r--. 1 npg npg 2.0G May  5 08:00 R1_50
-rw-rw-r--. 1 npg npg 2.0G May  5 08:02 R1_51
-rw-rw-r--. 1 npg npg 2.0G May  5 08:05 R1_52
-rw-rw-r--. 1 npg npg 2.0G May  5 08:07 R1_53
-rw-rw-r--. 1 npg npg 2.0G May  5 08:10 R1_54
-rw-rw-r--. 1 npg npg 2.0G May  5 08:13 R1_55
-rw-rw-r--. 1 npg npg 2.0G May  5 08:15 R1_56
-rw-rw-r--. 1 npg npg 2.0G May  5 08:18 R1_57
-rw-rw-r--. 1 npg npg 2.0G May  5 08:20 R1_58
-rw-rw-r--. 1 npg npg 2.0G May  5 08:23 R1_59
-rw-rw-r--. 1 npg npg 2.0G May  5 08:26 R1_60
-rw-rw-r--. 1 npg npg 2.0G May  5 08:28 R1_61
-rw-rw-r--. 1 npg npg 2.0G May  5 08:31 R1_62
-rw-rw-r--. 1 npg npg 2.0G May  5 08:34 R1_63
-rw-rw-r--. 1 npg npg 2.0G May  5 08:36 R1_64
-rw-rw-r--. 1 npg npg 2.0G May  5 08:39 R1_65
-rw-rw-r--. 1 npg npg 2.0G May  5 08:41 R1_66
-rw-rw-r--. 1 npg npg 1.6G May  5 08:44 R1_67
[npg@aidas-gsi ~]$ 

      Rate increased c. 06.15 
       multiple ASICs in multiple FEE64s hot HEC channels
      ASIC check load restores rates to those at beginning of May21/R1
      Switched Tape Server to storage mode - file May21/R1_67

08.53 All system wide checks OK *except* 

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida13 fault 	 0x62 : 	 0x9380 : 	 37662  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Most recent merger errors

MERGE Data Link (2497): bad timestamp  7 3 0xd1fc813c 0x073033d6 0x00001083773033d6 0x167c1083773033d6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ce7ff9 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1dd806e 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ed7fc1 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fd803c 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1cf8066 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1de8110 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ee8086 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fe7ff7 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1c38041 0x0e60a6d6 0x000010832e60a6d6 0x167c10832e60a6d6 0x167c108377304b46

09.04 FEE64 temps OK - attachment 2
      ADC data item stats - attachment 3
      1.8.L spectra - attachments 4 & 5
      1.8.W spectra - attachments 6 & 7
      Grafana DSSSD bias & Leakage current most recent 24h - attachment 8
      DSSSD bias & leakage current - attacment 9

11:32 R1 stopped to realign boards and check situation in S4
Attachment 1: 210504_2228_Stats.png
210504_2228_Stats.png
Attachment 2: Screenshot_2021-05-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-05-05_Statistics_aidas-gsi.png
Screenshot_2021-05-05_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(3).png
Attachment 8: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(4).png
Attachment 9: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-05_AIDA_Alerting_-_Grafana.png
Attachment 10: 400.png
400.png
  293   Tue May 4 14:26:50 2021 NHFEE64 Adapter wiring diagram and layout
A figure showing the FEE64 numbers (a previous diagram had 2/6 and 4/8 on the wrong side of the DSSD) and the wiring of pulser/HV
Also the jumpers connected to the boards

As always diagram is looking with direction of beam

Edit: Figure as signals (HV/Pulser) come from the beam right not beam left
Otherwise OK
Attachment 1: AIDA_Adapter_Configuration(1).png
AIDA_Adapter_Configuration(1).png
  292   Tue May 4 14:05:48 2021 NHMACB Cables
During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more
This may improve their connection and reduce the chance of WR errors seen in S460/S452
The connectors were all reseated a few times to clean the contacts too
Attachment 1: macb_front.jpg
macb_front.jpg
Attachment 2: macb_side.jpg
macb_side.jpg
  291   Tue May 4 13:53:22 2021 OH, TDTuesday 4 May
NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290

14.40 Rate spectra - attachment 1
      adc data items - attachment 2
       slow comparator 0xa
       stats for p+n much improved, n+n remain poor
      1.8.W spectra - attachments 3 & 4
      1.8.L spectra - attachments 5 & 6
       pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM

17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7
Attachment 1: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-05-04_Statistics_aidas-gsi.png
Screenshot_2021-05-04_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Attachment 7: 210504_Leakage.png
210504_Leakage.png
  290   Tue May 4 12:11:39 2021 OH TDBias test of triple
13:11 System wide checks all ok except *aida02 fails ADC calibration*
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue May 04 10:57:38 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

ASIC clocks have been synchronised and ADC re-calibrated

Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3

Detectors at 130V

Statistics - attachment 4

V-I Curve - attachment 5
Voltage	Ch0	Ch1
10	1.345	1.49
20	2.01	2.285
30	2.405	2.8
40	2.635	3.13
50	2.82	3.37
60	2.95	3.57
70	3.08	3.725
80	3.185	3.865
90	3.26	3.98
100	3.33	4.105
110	3.42	4.22
120	3.525	4.35
130	3.64	4.46
140	3.79	4.37
150	3.98	4.37
150+10min	3.9	4.485
Attachment 1: 210504_1325_Rate.png
210504_1325_Rate.png
Attachment 2: 210504_1326_Layout7.png
210504_1326_Layout7.png
Attachment 3: 210504_1327_Layout08.png
210504_1327_Layout08.png
Attachment 4: 210504_1334_Stats.png
210504_1334_Stats.png
Attachment 5: VICurve.png
VICurve.png
  289   Sat May 1 08:45:19 2021 TDAIDA 24cm x 8cm DSSSD + triPlast assembly for S496
Photos provided by Nic Hubbard and Helena Albers from the assembly of 2x triPlast
plus 2x MSL type BB18(DS)-1000 24cm x 8cm detector assembly - attachments 1-3.

Spacers installed between MSL type BB18(DS)-1000 24cm x 8cm PCBs - attachment 4.

AIDA support stand in 24cm x 8cm DSSSD mode ready for installation of snout assembly - attachments 5 & 6

AIDA 24cm x 8cm DSSSD plus 2x TriPlast installed in S4 - view downstream towards upstream window 
of AIDA detctor assembly - attachment 7

AIDA 24cm x 8cm DSSSD plus 2x TriPlast snout assembly installed at S4 - attachment 8

Side view of AIDA + triPLast + FATIMA + DEGAS - attachment 9
Attachment 1: MicrosoftTeams-image.png
MicrosoftTeams-image.png
Attachment 2: IMG_7649.JPG
IMG_7649.JPG
Attachment 3: IMG_7650.JPG
IMG_7650.JPG
Attachment 4: MicrosoftTeams-image_(1).png
MicrosoftTeams-image_(1).png
Attachment 5: image-7becce0a-c5bb-431b-8af9-597e2b8d3829.jpg
image-7becce0a-c5bb-431b-8af9-597e2b8d3829.jpg
Attachment 6: image-3ba4cdee-a8f4-48e4-b087-03ddf07504e5.jpg
image-3ba4cdee-a8f4-48e4-b087-03ddf07504e5.jpg
Attachment 7: image-beb082d4-a965-453f-8804-394ce65a2f70.jpg
image-beb082d4-a965-453f-8804-394ce65a2f70.jpg
Attachment 8: IMG_7697.JPG
IMG_7697.JPG
Attachment 9: IMG_7708.JPG
IMG_7708.JPG
Attachment 10: IMG_7682.JPG
IMG_7682.JPG
Attachment 11: IMG_7681.JPG
IMG_7681.JPG
Attachment 12: FATIMA_Triple_AIDA.png
FATIMA_Triple_AIDA.png
  288   Fri Apr 30 08:38:12 2021 TDFriday 30 April
09.40 All system wide checks OK *except*

FEE64 module aida07 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Thu Apr 29 17:50:06 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.44 DSSSD bias & leakage currents Ok - attachment 1
      Merger time seq errors - attachment 2
       no merger errors since restart yesterday
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      Merger/Tape Server server consoles - attachment 5
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 6

      DAQ continues OK - no data storage
Attachment 1: 310.png
310.png
Attachment 2: 311.png
311.png
Attachment 3: Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-30_Statistics_aidas-gsi.png
Screenshot_2021-04-30_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
  287   Thu Apr 29 16:56:42 2021 OHConfig changes for 16FEE
New set of ASIC settings produced 2021Apr29-13-16-00
- Note no longer odd/even for polarity of signal

New set of Options file for FEEs - Points to the new ASIC settings

New layouts for histograms grouped by side of the detector. Heavily weighted to the front side in terms of number of his

/MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 12 -l 16 -p 11001  &" to "./master64 -i 16 -l 16 -p 11001  &"

/MIDAS/DB/EXPERIMENTS/MERGE/Options/aidas-gsi/CONTENTS updated the number of links to 16.

System wide checks passed *except aida03 ADC calibration*

System left running to no storage.
  286   Wed Apr 28 17:39:35 2021 TDPreparations for setup of aida13-aida16
Updated /etc/dhcpd.conf

cd /etc
cp dhcpd.conf dhcpd.conf.BAK-280421

Add MAC addresses https://elog.ph.ed.ac.uk/DESPEC/285

aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed

New version of dhcpd.conf (and backup) - attachments 1 & 2

Cannot find dhcpd (and therefore re-read config and restart) in System - Administration -> Services
- has the dhcpd service been re-named or amalgamated with another service?



Updated /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl 

New version of startup.tcl (and backup) - attachments 3 & 4



Created rfs filesytems for aida13-aida24

[root@aidas-gsi]# cd /home/Embedded/XilinxLinux/ppc_4xx/rfs
[root@aidas-gsi rfs]# cd aida01
[root@aidas-gsi rfs]# tar cvf ../rfs.tar .

Create directories aida13-aida24, uid=10101, gid=npg, mode 777

[root@aidas-gsi rfs]# cd aida13; tar xvf ../rfs.tar .
 :
 :
etc

[root@aidas-gsi rfs]# ls -l 
total 1491880
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida01
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida02
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida03
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida04
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida05
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida06
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida07
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida08
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:35 aida09
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida10
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:34 aida11
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida12
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida13
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida14
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida15
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida16
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida17
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida18
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida19
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida20
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida21
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida22
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida23
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida24
-rw-r--r--.  1 root  root 1527582720 Apr 28 18:26 rfs.tar



Once dhcpd has been restarted it should be possible to power cycle all FEE64s, start system
and update firmware of newly installed FEE64s.
Attachment 1: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }

   host aida13 {
     hardware ethernet d8:80:39:42:d:15;
     fixed-address 192.168.11.13;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   }

   host aida14 {
     hardware ethernet d8:80:39:42:d:b;
     fixed-address 192.168.11.14;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   }

   host aida15 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.15
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15";
   }

   host aida16 {
     hardware ethernet d8:80:39:41:f6:ed;
     fixed-address 192.168.11.16;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida16";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



Attachment 2: dhcpd.conf.BAK-280421
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



Attachment 3: startup.tcl

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     variable ACQSERVER [first $ACQSERVERS]
    variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


Attachment 4: startup.tcl.BAK-280421

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
#     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     variable ACQSERVER [first $ACQSERVERS]
    variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


  285   Wed Apr 28 10:06:11 2021 NHWednesday 28 April
11.05 All system wide checks OK *except*
WR
	
		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc971 : 	 17  
aida02 fault 	 0x8bbf : 	 0x8bd0 : 	 17  
aida03 fault 	 0xae9e : 	 0xaeaf : 	 17  
aida04 fault 	 0x6e3 : 	 0x6f4 : 	 17  
aida05 fault 	 0x31a4 : 	 0x31bd : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x2303 : 	 25  
aida07 fault 	 0x438f : 	 0x43a8 : 	 25  
aida08 fault 	 0x61ed : 	 0x6206 : 	 25  
aida09 fault 	 0x1c32 : 	 0x1c40 : 	 14  
aida10 fault 	 0x9332 : 	 0x933f : 	 13  
aida11 fault 	 0x227b : 	 0x2288 : 	 13  
aida12 fault 	 0xb2fc : 	 0xb309 : 	 13  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA
	
			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x1 : 	 1  
aida02 fault 	 0x0 : 	 0x3 : 	 3  
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida11 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 4, Failed 8
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Looks like a big WR glitch happened at some point (master in issue?)

Memory
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     20     26     15      3      1      3      2      2      3      4   : 27848
aida02 :     25      7      1      4      1      4      2      2      3      3      6   : 36204
aida03 :     30      5      3      3      3      3      1      3      3      3      6   : 36464
aida04 :      5     10      0      5      2      3      1      3      3      3      6   : 36356
aida05 :     34     40     16     13      4      3      3      2      3      3      4   : 29160
aida06 :     16      8      4      2      1      4      1      3      3      3      6   : 36416
aida07 :     28     23     23     10      4      3      2      4      3      4      3   : 27736
aida08 :     26      9      3      3      1      3      1      3      3      3      6   : 36352
aida09 :     54     42     10     17      3      4      3      4      3      3      4   : 30376
aida10 :     17      5      7      4      3      3      3      3      2      3      6   : 35996
aida11 :     19     10     11      2      1      1      2      4      2      3      6   : 35916
aida12 :     82     42     12      7      4      2      3      2      2      3      4   : 27960


Options unchanged from ELOG 284

DAQ continues OK but New Merger window has disappeared... clearly still running in background!

DSSSD leakage currents decreasing as expected

-

Plan later today:

Prepare AIDA Kapton Cables for triple

install 4 new FEE64s into S4 (13,14,15,16)

Proposed layout of FEE64s (perspective with beam) in fig 5

Cards 9-12 will be moved on Friday when triple is installed

When ready MAC addressed will be given to OH/TD who will do the PC configuration including:
- Network IP allocation
- MIDAS registration
- Firmware updating
- Checks 

17.20:

The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water
No front-end adapter is connected yet
The MAC addresss are as follows
aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed
Attachment 1: elog_temps.png
elog_temps.png
Attachment 2: elog_gswr.png
elog_gswr.png
Attachment 3: elog_tse.png
elog_tse.png
Attachment 4: elog_hv.png
elog_hv.png
Attachment 5: AIDA_FEE.png.jpeg
AIDA_FEE.png.jpeg
  284   Tue Apr 27 08:26:01 2021 TDTuesday 27 April - aida09 system console reports 'received undefined information code: 1' and 'code: 9'
09.25 All system wide checks OK *except*


		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc963 : 	 3  
aida02 fault 	 0x8bbf : 	 0x8bc2 : 	 3  
aida03 fault 	 0xae9e : 	 0xaea1 : 	 3  
aida04 fault 	 0x6e3 : 	 0x6e6 : 	 3  
aida05 fault 	 0x31a4 : 	 0x31ac : 	 8  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f2 : 	 8  
aida07 fault 	 0x438f : 	 0x4397 : 	 8  
aida08 fault 	 0x61ed : 	 0x61f5 : 	 8  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     18     24     16      3      1      3      2      2      3      4   : 27792
aida02 :     25      6      3      5      1      4      2      2      3      3      6   : 36260
aida03 :     19     10      3      6      3      3      1      3      3      3      6   : 36556
aida04 :     21      7      3      4      2      4      2      2      3      3      6   : 36284
aida05 :     40     37     16     13      4      3      2      2      3      3      4   : 28904
aida06 :     32      4      4      3      0      4      1      3      3      3      6   : 36416
aida07 :     26     22     18     13      3      3      2      4      3      4      3   : 27672
aida08 :     15      9      4      1      2      3      1      3      3      3      6   : 36324
aida09 :     29     41     11     19      3      4      3      4      3      3      4   : 30348
aida10 :     17      5      8      4      3      3      3      3      2      3      6   : 36012
aida11 :     25      9      3      5      1      1      2      4      2      3      6   : 35900
aida12 :     80     43     12      7      4      2      3      2      2      3      4   : 27960



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

09.25 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
       aida09 zero rate
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK *except* aida09 - no data storage

10.00 from aida09 system console

24/03:04:39|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
24/03:04:39|Also in RDOGo_Operate: Enabled waveform readout
24/03:04:39|
24/03:04:39|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0
24/03:04:39|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A
24/03:04:39|WV:0:Buffer info. Length = 131072 : Offset = 0 
24/03:04:40|WV:1:Buffer info. Length = 131070 : Offset = 2 
24/03:04:40|WV:2:Buffer info. Length = 131068 : Offset = 4 
24/03:04:40|WV:3:Buffer info. Length = 131066 : Offset = 6 
24/03:04:40|WV:4:Buffer info. Length = 131064 : Offset = 8 
24/03:04:40|WV:5:Buffer info. Length = 131062 : Offset = 10 
24/03:04:40|WV:6:Buffer info. Length = 131060 : Offset = 12 
24/03:04:40|WV:7:Buffer info. Length = 131058 : Offset = 14 
24/03:04:40|WV:8:Buffer info. Length = 131056 : Offset = 16 
24/03:04:40|WV:9:Buffer info. Length = 131054 : Offset = 18 
24/03:04:40|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:40|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000 
24/03:04:40|Last Data : Databuffer  274139128 : 0 => C01B7F88 : 1 => 0C05B135 
24/03:04:40| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:40| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:40| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:41| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:41| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:41| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:41| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:41| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:41|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455 
24/03:04:41|Last Data : Databuffer  274139128 : 0 => C0127D9D : 1 => 0DB9DEED 
24/03:04:41| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:41| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:41| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:42| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:42| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:42| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:42| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:42| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:42|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:42|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED 
24/03:04:42|Last Data : Databuffer  274139128 : 0 => C0217F73 : 1 => 0F816B3D 
24/03:04:42| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:42| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:42| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:43| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:43| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:43| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:43| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:43| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:43|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95 
24/03:04:43|Last Data : Databuffer  274139128 : 0 => C0167F53 : 1 => 0148F2DD 
24/03:04:43| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:43| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:43| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:44| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:44| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:44| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:44| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:44|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD 
24/03:04:44|Last Data : Databuffer  274139128 : 0 => C0177F46 : 1 => 047FF0CD 
24/03:04:44| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:44| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:44| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:45| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:45| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:45| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:45| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:47|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:56|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:05|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:05:15|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:24|197 spectra zeroed
25/10:21:10|do_GetState returned z=0 and 1
25/10:21:11|Data Acquisition Statistics counters now cleared
25/10:24:01|Clear Statistics (1)
25/10:24:01|bad event data (0x45000034 0x84e64000)
26/16:52:50|bad event data (0x40061e21 0xc0a80b63)
26/16:52:50|received undefined information code: 1
26/16:52:50|bad event data (0x0101080a 0xe9b68def)
26/16:52:50|bad event data (0x03527827 0x0101050a)
26/16:52:50|received undefined information code: 9

From system console

Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
Apr 24 02:52:42 aida09 kernel: IP-Config: Complete:
Apr 24 02:52:42 aida09 kernel:      device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
Apr 24 02:52:42 aida09 kernel:      host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
Apr 24 02:52:42 aida09 kernel:      bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99
Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init
Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000
Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services
Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253
Apr 24 02:53:04 aida09 kernel: xaida: open:
Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open:
Apr 24 02:57:33 aida09 kernel: xaida: open:
Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99
Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec)
Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99
Attachment 1: 303.png
303.png
Attachment 2: Screenshot_2021-04-27_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-27_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-27_Statistics_aidas-gsi.png
Screenshot_2021-04-27_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-27_GSI_White_Rabbit_Timestamp_Control_aidas-gsi.png
Screenshot_2021-04-27_GSI_White_Rabbit_Timestamp_Control_aidas-gsi.png
Attachment 5: Screenshot_2021-04-27_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-27_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-27_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-27_AIDA_Alerting_-_Grafana.png
  283   Mon Apr 26 09:41:21 2021 TDMonday 26 April
10.42 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc962 : 	 2  
aida02 fault 	 0x8bbf : 	 0x8bc1 : 	 2  
aida03 fault 	 0xae9e : 	 0xaea0 : 	 2  
aida04 fault 	 0x6e3 : 	 0x6e5 : 	 2  
aida05 fault 	 0x31a4 : 	 0x31ab : 	 7  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f1 : 	 7  
aida07 fault 	 0x438f : 	 0x4396 : 	 7  
aida08 fault 	 0x61ed : 	 0x61f4 : 	 7  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25      8     28      8      3      2      2      3      2      3      4   : 27940
aida02 :     24      9      3      5      1      3      1      3      3      3      6   : 36408
aida03 :     24      6      5      4      3      3      1      3      3      3      6   : 36512
aida04 :     23      6      6      1      3      3      1      3      3      3      6   : 36428
aida05 :     15     21     27      7      4      2      2      3      3      3      4   : 29044
aida06 :     24      8      2      4      1      3      1      3      3      3      6   : 36352
aida07 :     29     24     28      9      3      3      2      2      2      3      4   : 27732
aida08 :     17     10      1      1      1      4      1      3      3      3      6   : 36356
aida09 :     41     49     18     14      3      4      3      4      2      3      4   : 29388
aida10 :      3      1      6      4      2      3      3      3      2      3      6   : 35828
aida11 :     22      7      5      4      2      1      2      4      2      3      6   : 35936
aida12 :     65     30     23      9      3      1      2      3      2      3      4   : 28100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021


10.47 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK - no data storage
Attachment 1: 302.png
302.png
Attachment 2: Screenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-26_Statistics_aidas-gsi.png
Screenshot_2021-04-26_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-26_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-26_AIDA_Alerting_-_Grafana.png
  282   Sun Apr 25 10:22:16 2021 TDSunday 25 April
11.25 All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0xc95f : 	 0xc960 : 	 1  
aida02 fault 	 0x8bbe : 	 0x8bbf : 	 1  
aida03 fault 	 0xae9d : 	 0xae9e : 	 1  
aida04 fault 	 0x6e2 : 	 0x6e3 : 	 1  
aida05 fault 	 0x31a2 : 	 0x31a4 : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22ea : 	 3  
aida07 fault 	 0x438d : 	 0x438f : 	 2  
aida08 fault 	 0x61eb : 	 0x61ed : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     55     21     31      6      3      2      2      3      2      3      4   : 28148
aida02 :     28      3      5      4      0      3      1      3      3      3      6   : 36312
aida03 :     25     10      2      2      2      4      1      3      3      3      6   : 36500
aida04 :     14      7     10      1      2      2      2      3      3      3      6   : 36528
aida05 :     58     35     28      6      4      2      2      3      3      3      4   : 29312
aida06 :     16     11      6      1      0      4      1      3      3      3      6   : 36376
aida07 :     75     31     25      9      4      3      2      2      2      3      4   : 27988
aida08 :     28      7      1      1      1      4      1      3      3      3      6   : 36376
aida09 :     50     48     28     10      3      3      3      4      2      3      4   : 29320
aida10 :     26      5      4      3      1      3      2      4      2      3      6   : 36080
aida11 :     21      7      5      3      5      2      2      4      2      3      6   : 36220
aida12 :     64     29     26      7      3      1      2      3      2      3      4   : 28072

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors reported - server terminal now reset & clear'd

MERGE Data Link (4408): bad timestamp  10 3 0xc2b67d8f 0x0b50c3e2 0x0000f437cb50c3e2 0x1678f437cb50c3e2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2877f27 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a07fcd 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77c9c 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc28b7d9c 0x0b50d382 0x0000f437cb50d382 0x1678f437cb50d382 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2937e6c 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a17d78 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2817fff 0x0b50eaf2 0x0000f437cb50eaf2 0x1678f437cb50eaf2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77ecf 0x06fb3d52 0x0000f437cb50eaf2 0x1678f437c6fb3d52 0x1678f43a16f1e6b2

11.25 Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 1
      DSSSD bias |& leakage currents Ok - attachment 2
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      merger time seq errors - attachment 5

      All histograms, stats and merge stats zero'd
      WR decoder errors *new*  baseline

11.45 1.8.L spectra - attachments 6 & 7
      peak width FWHM aida05 65 ch, aida06 95 ch
      1.8.W spectra = attachments 8 & 9
Attachment 1: Screenshot_2021-04-25_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-25_AIDA_Alerting_-_Grafana.png
Attachment 2: 300.png
300.png
Attachment 3: Screenshot_2021-04-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-25_Statistics_aidas-gsi.png
Screenshot_2021-04-25_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-25_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-25_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(3).png
  281   Sat Apr 24 15:48:38 2021 TDaida12 crash - 08:33:03 Monday 19 April
from ttyUSB5 system console log

19/07:25:59|page:c047b060 flags:(null) count:-1 mapcount:0 mapping:(null) index:0
19/07:25:59|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:00|Faulting instruction address: 0xc006e814
19/07:26:00|Oops: Kernel access of bad area, sig: 11 [#1]
19/07:26:00|PREEMPT Xilinx Virtex440
19/07:26:00|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:00|NIP: c006e814 LR: c006e78c CTR: c026327c
19/07:26:00|REGS: c60f3860 TRAP: 0300   Not tainted  (2.6.31)
19/07:26:00|MSR: 00021000 <ME,CE>  CR: 28222024  XER: 20000000
19/07:26:00|DEAR: 00100104, ESR: 00800000
19/07:26:00|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:00|GPR00: 00000000 c60f3910 c6b0a0a0 c038c3a8 00000000 000001a5 c047b078 00000041
19/07:26:00|GPR08: 00000001 00100100 00000004 c038c3a8 48282028 2015881c 00000001 00000000
19/07:26:00|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c60f2000
19/07:26:00|GPR24: 00000000 00000001 00029000 c038c384 00000001 c047b060 c038c39c c038c394
19/07:26:00|NIP [c006e814] get_page_from_freelist+0x188/0x4cc
19/07:26:00|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:00|Call Trace:
19/07:26:00|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:00|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:01|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:01|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:01|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:01|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:01|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:01|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:01|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:01|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:01|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:01|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:01|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:01|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:01|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:01|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:01|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:01|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:01|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:01|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:01|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:01|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:02|Instruction dump:
19/07:26:02|2f890000 419e0008 7c004a2c 38dd0018 7f861800 409effd8 7f861800 419e02e0
19/07:26:02|813d0018 81660004 7f3ccb78 912b0000 <91690004> 92dd0018 92a60004 813e0000
19/07:26:02|---[ end trace 87f7fdd1c04ff304 ]---
19/07:26:02|note: syslogd[224] exited with preempt_count 1
19/07:26:02|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:02|Faulting instruction address: 0xc006e388
19/07:26:02|Oops: Kernel access of bad area, sig: 11 [#2]
19/07:26:02|PREEMPT Xilinx Virtex440
19/07:26:02|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:02|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:02|REGS: c60f35d0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:02|MSR: 00021000 <ME,CE>  CR: 28222082  XER: 00000000
19/07:26:02|DEAR: 00100104, ESR: 00800000
19/07:26:02|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:02|GPR00: 00000000 c60f3680 c6b0a0a0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:02|GPR08: 00000008 c03c7738 00000006 00100100 00000000 2015881c c69e63f8 1fd84000
19/07:26:02|GPR16: c0390000 c60f2000 c69e63f8 c6ba2128 c6178740 c036f924 ffffffff 00000000
19/07:26:03|GPR24: 1fd84000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7720
19/07:26:03|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:03|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:03|Call Trace:
19/07:26:03|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:03|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:03|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:03|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:03|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:03|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:03|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:03|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:03|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:03|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:03|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:03|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:03|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:03|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:03|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:04|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:04|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:04|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:04|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:04|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:04|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:04|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:04|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:04|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:04|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:04|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:04|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:04|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:04|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:04|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:04|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:04|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:04|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:04|Instruction dump:
19/07:26:04|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:05|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:05|---[ end trace 87f7fdd1c04ff305 ]---
19/07:26:05|Fixing recursive fault but reboot is needed!
19/07:26:05|BUG: scheduling while atomic: syslogd/224/0x00000006
19/07:26:05|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:05|Call Trace:
19/07:26:05|[c60f34c0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:05|[c60f3500] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:05|[c60f3510] [c02a8020] schedule+0x2f4/0x34c
19/07:26:05|[c60f3550] [c00382a8] do_exit+0x544/0x61c
19/07:26:05|[c60f3590] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:05|[c60f35b0] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:05|[c60f35c0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:05|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:05|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:05|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:05|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:05|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:05|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:05|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:06|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:06|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:06|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:06|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:06|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:06|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:06|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:06|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:06|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:06|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:06|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:06|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:06|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:06|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:06|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:06|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:06|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:06|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:06|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:06|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:07|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:07|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:07|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:07|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:07|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:07|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:07|Unable to handle kernel paging request for data at address 0x00100100
19/07:26:07|Faulting instruction address: 0xc006e9e8
19/07:26:07|Oops: Kernel access of bad area, sig: 11 [#3]
19/07:26:07|PREEMPT Xilinx Virtex440
19/07:26:07|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:07|NIP: c006e9e8 LR: c006e78c CTR: c0285b38
19/07:26:07|REGS: c61adab0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:07|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:07|DEAR: 00100100, ESR: 00000000
19/07:26:07|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:07|GPR00: 00000010 c61adb60 c6b0a4c0 c038c3a8 00000000 000001a5 00000000 00000041
19/07:26:07|GPR08: 00000001 00100100 00000004 c61ac000 22024048 2079d364 00000001 00000000
19/07:26:08|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c61ac000
19/07:26:08|GPR24: 00000000 00000001 00029000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:26:08|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:08|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:08|Call Trace:
19/07:26:08|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:08|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:08|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:08|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:08|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:08|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:08|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:08|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:08|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:08|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:08|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:08|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:08|Instruction dump:
19/07:26:08|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:09|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:09|---[ end trace 87f7fdd1c04ff306 ]---
19/07:26:09|note: klogd[227] exited with preempt_count 1
19/07:26:09|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:09|Faulting instruction address: 0xc006e388
19/07:26:09|Oops: Kernel access of bad area, sig: 11 [#4]
19/07:26:09|PREEMPT Xilinx Virtex440
19/07:26:09|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:09|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:09|REGS: c61ad820 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:09|MSR: 00021000 <ME,CE>  CR: 22224082  XER: 00000000
19/07:26:09|DEAR: 00100104, ESR: 00800000
19/07:26:09|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:09|GPR00: 00000000 c61ad8d0 c6b0a4c0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:09|GPR08: 00000008 c03c7e98 00000006 00100100 00000000 2079d364 c6a5640c 20739000
19/07:26:09|GPR16: c0390000 c61ac000 c6a5640c c60a6320 c6178c80 c036f924 ffffffff 00000000
19/07:26:09|GPR24: 20739000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7e80
19/07:26:10|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:10|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:10|Call Trace:
19/07:26:10|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:10|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:10|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:10|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:10|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:10|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:10|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:10|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:10|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:10|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:10|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:10|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:10|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:10|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:10|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:10|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:10|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:11|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:11|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:11|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:11|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:11|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:11|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:11|Instruction dump:
19/07:26:11|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:11|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:11|---[ end trace 87f7fdd1c04ff307 ]---
19/07:26:11|Fixing recursive fault but reboot is needed!
19/07:26:11|BUG: scheduling while atomic: klogd/227/0x00000006
19/07:26:11|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:11|Call Trace:
19/07:26:11|[c61ad710] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:11|[c61ad750] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:11|[c61ad760] [c02a8020] schedule+0x2f4/0x34c
19/07:26:11|[c61ad7a0] [c00382a8] do_exit+0x544/0x61c
19/07:26:11|[c61ad7e0] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:11|[c61ad800] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61ad810] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:12|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:12|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:12|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:12|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:12|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:12|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:12|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:12|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:12|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:12|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:12|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:12|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:12|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:12|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:12|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:13|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:13|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:13|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:13|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:13|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:13|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:13|Faulting instruction address: 0xc006e9e8
19/07:26:13|Oops: Kernel access of bad area, sig: 11 [#5]
19/07:26:13|PREEMPT Xilinx Virtex440
19/07:26:13|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:13|NIP: c006e9e8 LR: c006e78c CTR: c006b2c4
19/07:26:13|REGS: c6349900 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:13|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:13|DEAR: 00000000, ESR: 00000000
19/07:26:13|TASK = c62fa500[379] 'AidaExecV9' THREAD: c6348000
19/07:26:13|GPR00: 00000009 c63499b0 c62fa500 c038c3a8 00000000 000001a5 00100100 00000041
19/07:26:13|GPR08: 00000001 00000000 00000004 c6348000 22022048 1005520c 00000001 00000000
19/07:26:13|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6348000
19/07:26:14|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 ffffffe8 c038c39c c038c394
19/07:26:14|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:14|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:14|Call Trace:
19/07:26:14|[c63499b0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:14|[c6349a40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:14|[c6349ac0] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:14|[c6349b20] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:26:14|[c6349b40] [c006b2e0] mempool_alloc_slab+0x1c/0x2c
19/07:26:14|[c6349b50] [c006b5d8] mempool_alloc+0x5c/0x13c
19/07:26:14|[c6349ba0] [c0294600] rpc_malloc+0x34/0x9c
19/07:26:14|[c6349bc0] [c028da44] call_allocate+0xc4/0x194
19/07:26:14|[c6349be0] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:14|[c6349c20] [c028d120] rpc_run_task+0x44/0x90
19/07:26:14|[c6349c30] [c028d234] rpc_call_sync+0x54/0x84
19/07:26:14|[c6349c60] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
19/07:26:14|[c6349c90] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
19/07:26:14|[c6349d50] [c0129b3c] nfs_permission+0x178/0x1c4
19/07:26:14|[c6349d70] [c00a0bf4] inode_permission+0x94/0xbc
19/07:26:15|[c6349d80] [c00a2bc0] __link_path_walk+0xb0/0xf7c
19/07:26:15|[c6349df0] [c00a3ad4] path_walk+0x48/0xc8
19/07:26:15|[c6349e20] [c00a3c44] do_path_lookup+0x64/0x6c
19/07:26:15|[c6349e40] [c00a4b14] do_filp_open+0xc8/0x914
19/07:26:15|[c6349f10] [c0095314] do_sys_open+0x64/0x130
19/07:26:15|[c6349f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:15|Instruction dump:
19/07:26:15|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:15|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:15|---[ end trace 87f7fdd1c04ff308 ]---
19/07:26:15|note: AidaExecV9[379] exited with preempt_count 1
19/07:26:15|Call Trace:
19/07:26:15|[c6311b50] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:15|[c6311b90] [c006da9c] bad_page+0x94/0x12c
19/07:26:15|[c6311bb0] [c006eadc] get_page_from_freelist+0x450/0x4cc
19/07:26:15|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:15|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:15|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:15|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:15|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:16|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:16|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:16|Faulting instruction address: 0xc006e9e8
19/07:26:16|Oops: Kernel access of bad area, sig: 11 [#6]
19/07:26:16|PREEMPT Xilinx Virtex440
19/07:26:16|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:16|NIP: c006e9e8 LR: c006e78c CTR: c01c13cc
19/07:26:16|REGS: c6311b00 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:16|MSR: 00021000 <ME,CE>  CR: 24222024  XER: 00000000
19/07:26:16|DEAR: 00000000, ESR: 00000000
19/07:26:16|TASK = c62fb9a0[375] 'AidaExecV9' THREAD: c6310000
19/07:26:16|GPR00: 00000009 c6311bb0 c62fb9a0 c038c3a8 c62fb9a0 00000010 00100100 0000594b
19/07:26:16|GPR08: 000000a0 00000000 00000000 c6310000 24222042 1005520c 00000001 00000000
19/07:26:16|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6310000
19/07:26:16|GPR24: 00000000 00000001 00029000 c038c384 00000001 ffffffe8 c038c39c c038c394
19/07:26:16|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:16|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:17|Call Trace:
19/07:26:17|[c6311bb0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:17|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:17|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:17|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:17|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:17|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:17|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:17|Instruction dump:
19/07:26:17|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:17|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:17|---[ end trace 87f7fdd1c04ff309 ]---
19/07:26:17|note: AidaExecV9[375] exited with preempt_count 1
19/07:26:17|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:18|Faulting instruction address: 0xc006e388
19/07:26:18|Oops: Kernel access of bad area, sig: 11 [#7]
19/07:26:18|PREEMPT Xilinx Virtex440
19/07:26:18|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:18|NIP: c006e388 LR: c006e348 CTR: 00000000
19/07:26:18|REGS: c6835e10 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:18|MSR: 00021000 <ME,CE>  CR: 24000082  XER: 00000000
19/07:26:18|DEAR: 00100104, ESR: 00800000
19/07:26:18|TASK = c6824880[5] 'events/0' THREAD: c6834000
19/07:26:18|GPR00: 00000000 c6835ec0 c6824880 00000000 00000003 c038c3a8 ffffffff c0498000
19/07:26:18|GPR08: 00000008 c0419a78 00000027 00100100 002515ba 00005aa8 bfffffff 5fffffff
19/07:26:18|GPR16: ffff7fd7 fffeffaf 00120110 7ffefd7f c031aef4 c02fb040 c02fb008 00000002
19/07:26:18|GPR24: c6834000 00000000 c038c384 c038c384 00000000 00029000 c038c39c c0419a60
19/07:26:18|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:18|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:18|Call Trace:
19/07:26:18|[c6835ec0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:18|[c6835ef0] [c00915c4] kmem_freepages+0xd8/0x134
19/07:26:19|[c6835f00] [c0091ab8] slab_destroy+0x34/0x8c
19/07:26:19|[c6835f10] [c0091e20] drain_freelist+0x9c/0x110
19/07:26:19|[c6835f40] [c0093158] cache_reap+0x104/0x138
19/07:26:19|[c6835f60] [c0048908] worker_thread+0x140/0x204
19/07:26:19|[c6835fc0] [c004cfc8] kthread+0x78/0x7c
19/07:26:19|[c6835ff0] [c000e140] kernel_thread+0x4c/0x68
19/07:26:19|Instruction dump:
19/07:26:19|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:19|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:19|---[ end trace 87f7fdd1c04ff30a ]---
19/07:26:19|note: events/0[5] exited with preempt_count 1
19/07:26:19|Unable to handle kernel paging request for data at address 0x00100100
19/07:29:19|Faulting instruction address: 0xc006e9e8
19/07:29:19|Oops: Kernel access of bad area, sig: 11 [#8]
19/07:29:19|PREEMPT Xilinx Virtex440
19/07:29:19|Modules linked in: aidamem xdriver xh_spidev_register
19/07:29:19|NIP: c006e9e8 LR: c006e78c CTR: c025c424
19/07:29:19|REGS: c0391a60 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:29:19|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:29:19|DEAR: 00100100, ESR: 00000000
19/07:29:19|TASK = c036e318[0] 'swapper' THREAD: c0390000
19/07:29:19|GPR00: 00000010 c0391b10 c036e318 c038c3a8 00000000 000001a5 00000000 00000041
19/07:29:19|GPR08: 00000001 00100100 00000004 c0390000 22022048 00005aa8 00000001 00000000
19/07:29:19|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c0390000
19/07:29:19|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:29:19|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:29:19|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:19|Call Trace:
19/07:29:19|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:29:20|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:20|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:20|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:20|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:20|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:20|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:20|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:20|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:20|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:20|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:20|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:20|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:20|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:20|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:20|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:20|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:20|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:20|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:20|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:21|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:21|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:21|Instruction dump:
19/07:29:21|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:29:21|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:29:21|Kernel panic - not syncing: Fatal exception in interrupt
19/07:29:21|Call Trace:
19/07:29:21|[c0391990] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:29:21|[c03919d0] [c00345bc] panic+0x94/0x168
19/07:29:21|[c0391a20] [c000bd44] die+0x178/0x18c
19/07:29:21|[c0391a40] [c001191c] bad_page_fault+0x90/0xd8
19/07:29:21|[c0391a50] [c000e834] handle_page_fault+0x7c/0x80
19/07:29:21|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:21|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:21|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:21|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:21|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:21|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:21|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:22|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:22|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:22|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:22|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:22|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:22|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:22|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:22|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:22|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:22|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:22|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:22|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:22|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:22|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:22|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:22|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
19/07:32:23|
Found 0 errors checking kernel image
19/07:32:24|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
19/07:32:24|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/07:32:24|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/07:32:24|SPIconfig done now to set up the LMK3200 registers
19/07:32:24|LMK #0 : regInit[0]=0x80000000
19/07:32:24|LMK #0 : regInit[1]=0x10070600
19/07:32:24|LMK #0 : regInit[2]=0x60601
19/07:32:24|LMK #0 : regInit[3]=0x60602
19/07:32:25|LMK #0 : regInit[4]=0x60603
19/07:32:25|LMK #0 : regInit[5]=0x70624
19/07:32:25|LMK #0 : regInit[6]=0x70605
19/07:32:25|LMK #0 : regInit[7]=0x70606
19/07:32:25|LMK #0 : regInit[8]=0x70627
19/07:32:25|LMK #0 : regInit[9]=0x10000908
19/07:32:25|LMK #0 : regInit[10]=0xA0022A09
19/07:32:25|LMK #0 : regInit[11]=0x82800B
19/07:32:25|LMK #0 : regInit[12]=0x28C800D
19/07:32:25|LMK #0 : regInit[13]=0x830020E
19/07:32:25|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/07:32:25|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
19/07:32:25|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/07:32:25|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
19/07:32:26|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/07:32:29|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xcf 0xac
19/07:32:35|Using Xilinx Virtex440 machine description
19/07:32:36|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/07:32:36|Zone PFN ranges:
19/07:32:36|  DMA      0x00000000 -> 0x00007000
19/07:32:36|  Normal   0x00007000 -> 0x00007000
19/07:32:36|Movable zone start PFN for each node
19/07:32:36|early_node_map[1] active PFN ranges
19/07:32:36|    0: 0x00000000 -> 0x00007000
19/07:32:36|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/07:32:36|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/07:32:36|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/07:32:36|PID hash table entries: 512 (order: 9, 2048 bytes)
19/07:32:36|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/07:32:37|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/07:32:37|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/07:32:37|Kernel virtual memory layout:
19/07:32:37|  * 0xffffe000..0xfffff000  : fixmap
19/07:32:37|  * 0xfde00000..0xfe000000  : consistent mem
19/07:32:37|  * 0xfde00000..0xfde00000  : early ioremap
19/07:32:37|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/07:32:37|NR_IRQS:512
19/07:32:37|clocksource: timebase mult[a00000] shift[22] registered
19/07:32:37|Console: colour dummy device 80x25
19/07:32:37|Mount-cache hash table entries: 512
19/07:32:37|NET: Registered protocol family 16
19/07:32:37|PCI: Probing PCI hardware
19/07:32:37|bio: create slab <bio-0> at 0
19/07:32:37|NET: Registered protocol family 2
19/07:32:37|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/07:32:37|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/07:32:37|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/07:32:37|TCP: Hash tables configured (established 4096 bind 4096)
19/07:32:37|TCP reno registered
19/07:32:37|NET: Registered protocol family 1
19/07:32:37|ROMFS MTD (C) 2007 Red Hat, Inc.
19/07:32:38|msgmni has been set to 214
19/07:32:38|io scheduler noop registered
19/07:32:38|io scheduler anticipatory registered
19/07:32:38|io scheduler deadline registered
19/07:32:38|io scheduler cfq registered (default)
19/07:32:38|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/07:32:38|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/07:32:38|console [ttyS0] enabled
19/07:32:38|brd: module loaded
19/07:32:38|loop: module loaded
19/07:32:38|Device Tree Probing 'ethernet'
19/07:32:38|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:cf:ac
19/07:32:38|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/07:32:38|XLlTemac: DCR address: 0x80
19/07:32:38|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/07:32:38|XLlTemac: Allocating DMA descriptors with kmalloc
19/07:32:38|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/07:32:38|XTemac: PHY detected at address 7.
19/07:32:38|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/07:32:38|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39|Using buffer write method
19/07:32:39|cfi_cmdset_0001: Erase suspend on write enabled
19/07:32:39|cmdlinepart partition parsing not available
19/07:32:39|RedBoot partition parsing not available
19/07:32:39|Creating 5 MTD partitions on "fc000000.flash":
19/07:32:39|0x000000000000-0x000000500000 : "golden_firmware"
19/07:32:39|0x000000500000-0x000000800000 : "golden_kernel"
19/07:32:39|0x000000800000-0x000000d00000 : "user_firmware"
19/07:32:39|0x000000d00000-0x000000fe0000 : "user_kernel"
19/07:32:39|0x000000fe0000-0x000001000000 : "env_variables"
19/07:32:39|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/07:32:39|SPI: XIlinx spi: bus number now 32766
19/07:32:39|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/07:32:39|SPI: XIlinx spi: bus number now 32765
19/07:32:40|mice: PS/2 mouse device common for all mice
19/07:32:40|Device Tree Probing 'i2c'
19/07:32:40| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/07:32:40|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/07:32:40|TCP cubic registered
19/07:32:40|NET: Registered protocol family 17
19/07:32:40|RPC: Registered udp transport module.
19/07:32:40|RPC: Registered tcp transport module.
19/07:32:40|eth0: XLlTemac: Options: 0x3fa
19/07:32:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/07:32:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/07:32:40|eth0: XLlTemac: speed set to 1000Mb/s
19/07:32:43|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/07:32:43|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/07:32:43|Sending DHCP requests ., OK
19/07:32:44|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.12
19/07:32:44|IP-Config: Complete:
19/07:32:44|     device=eth0, addr=192.168.11.12, mask=255.255.255.0, gw=255.255.255.255,
19/07:32:44|     host=aida12, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/07:32:44|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12
19/07:32:44|Looking up port of RPC 100003/2 on 192.168.11.99
19/07:32:44|Looking up port of RPC 100005/1 on 192.168.11.99
19/07:32:44|VFS: Mounted root (nfs filesystem) on device 0:12.
19/07:32:44|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/07:32:45|Starting sysinit...
19/07:32:45|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/07:32:46|            Press 'I' to enter interactive startup.
19/07:32:46|Setting clock  (utc): Mon Apr 19 07:32:47 BST 2021 [  OK  ]
19/07:32:47|Building the cache [  OK  ]
19/07:32:47|Setting hostname aida12:  [  OK  ]
19/07:32:48|Mounting local filesystems:  [  OK  ]
19/07:32:50|Enabling /etc/fstab swaps:  [  OK  ]
19/07:32:53|Finishing sysinit...
INIT: Entering runlevel: 3
19/07:32:56|Entering non-interactive startup
19/07:32:56|FATAL: Module ipv6 not found.
19/07:32:57|Bringing up loopback interface:  [  OK  ]
19/07:32:59|FATAL: Module ipv6 not found.
19/07:33:00|Starting system logger: [  OK  ]
19/07:33:01|Starting kernel logger: [  OK  ]
19/07:33:01|Starting rpcbind: [  OK  ]
19/07:33:03|Mounting NFS filesystems:  [  OK  ]
19/07:33:03|Mounting other filesystems:  [  OK  ]
19/07:33:04|Starting xinetd: [  OK  ]
19/07:33:05|Starting midas:  Starting MIDAS Data Acquisition for aida12
19/07:33:05|xaida: device parameters: base=0x81000000 size=0x200000
19/07:33:12|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/07:33:12|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/07:33:12|xaida: driver assigned major number 254
19/07:33:12|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/07:33:17|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/07:33:17|AIDAMEM: aidamem: driver assigned major number 253
19/07:33:17|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/07:33:24|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/07:33:24|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/07:33:24|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/07:33:24|Computer Name = aida12; Temp Directory = /tmp/tcl361
19/07:33:31|
19/07:33:34|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  280   Sat Apr 24 15:28:17 2021 TDaida05 crash - 11:08:07 Tuesday 20 April
From ttyUSB7 system console log

20/09:59:30|get_ASICBlk : Blk=1 : bytes = 793600 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0017F1D : 1 => 0FAC4BAF
20/09:59:31|Last Data : Databuffer  273884152 : 0 => 80426EF4 : 1 => 0BD70000
20/09:59:31| C03E7F21 : 03FCCB77 | C03D7EF5 : 03FCEE9F | C0047DA3 : 03FCF5A7 | C0207E21 : 03FCF5A7
20/09:59:31| C0317E77 : 03FCF5A7 | C0087E6D : 03FCF66F | C0107E98 : 03FCF66F | C0217ED8 : 03FCF66F
20/09:59:31| C0327E19 : 03FCF66F | C00B7DF0 : 03FCF737 | C0117DD5 : 03FCF737 | C0227F27 : 03FCF737
20/09:59:31| C03C7F23 : 03FCF737 | C0137EF9 : 03FCF7FF | C0297EB7 : 03FCF7FF | C03E7EC7 : 03FCF7FF
20/09:59:31| C0167DFE : 03FCF8C7 | C02A7FC3 : 03FCF8C7 | C03F7E82 : 03FCF8C7 | C01B7E91 : 03FCF98F
20/09:59:31| C02F7E8F : 03FCF98F | C01D7DC4 : 03FCFA57 | C01E7D40 : 03FCFB1F | 80426EF3 : 03FD0000
20/09:59:32| 8050023F : 03FD0000 | C03D7F1A : 03FD303F | 80426EF3 : 03FD4000 | 8050023F : 03FD4000
20/09:59:32| C0317F2F : 03FD5047 | C0047DF2 : 03FD510F | C01B7EAD : 03FD54F7 | C03D7F09 : 03FD5D8F
20/09:59:32|SIGPIPE ERROR - Error in xfer ASIC procedures: location=0
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_bxaida: release:
20/10:00:41|loAIDAMEM: aidamem_release:
20/10:00:41|ck busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|SIGSEGV ERROR - Error in xfer ASIC procedures: location=0
20/10:00:41|xfer_block busy
20/10:00:41|do_GetState returned z=0 and 1
20/10:01:27|executing generic doStop
20/10:02:41|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:02:42|StopC:
20/10:02:42|do_GetState returned z=0 and 6
20/10:02:47|do_GetState returned z=0 and 6
20/10:02:58|do_GetState returned z=0 and 6
20/10:03:10|do_GetState returned z=0 and 6
20/10:03:21|do_GetState returned z=0 and 6
20/10:03:33|do_GetState returned z=0 and 6
20/10:03:44|do_GetState returned z=0 and 6
20/10:03:56|do_GetState returned z=0 and 6
20/10:04:07|do_GetState returned z=0 and 6
20/10:04:19|do_GetState returned z=0 and 6
20/10:04:30|do_GetState returned z=0 and 6
20/10:04:42|do_GetState returned z=0 and 6
20/10:04:53|do_GetState returned z=0 and 6
20/10:05:04|do_GetState returned z=0 and 6
20/10:05:16|do_GetState returned z=0 and 6
20/10:05:27|do_GetState returned z=0 and 6
20/10:05:39|do_GetState returned z=0 and 6
20/10:05:50|do_GetState returned z=0 and 6
20/10:06:02|do_GetState returned z=0 and 6
20/10:06:12|executing generic doReset
20/10:06:21|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:06:22|ResetC:
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:34|do_GetState returned z=0 and 6
20/10:06:45|do_GetState returned z=0 and 6
20/10:06:57|do_GetState returned z=0 and 6
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
20/10:07:15|[  OK  ]
20/10:07:15|Stopping rpcbind: [  OK  ]
20/10:07:17|Shutting down kernel logger: [  OK  ]
20/10:07:19|Shutting down system logger: [  OK  ]
20/10:07:19|Stopping xinetd: [  OK  ]
20/10:07:19|[  OK  ]
20/10:07:19|Sending all processes the TERM signal...
20/10:07:20|xaida: release:
20/10:07:20|Sending all processes the KILL signal...
20/10:07:25|Saving random seed:
20/10:07:25|Syncing hardware clock to system time xaida: open:
20/10:07:25|xaida: release:
20/10:07:25|xaida: open:
20/10:07:25|xaida: ioctl: called with command 536899587
20/10:07:25|xaida: ioctl: called with command 1076129801
20/10:07:25|RTCxaida: release:
20/10:07:25|_RD_TIME: Inappropriate ioctl for device
20/10:07:25|ioctl() to /dev/rtc to read the time failed.
20/10:07:25|
20/10:07:25|Please stand by while rebooting the system...
Sj&#65533;5õÅ07:27|Restarting system.
20/10:07:29|ÉU&#65533;×®­Ë
                   R&#65533;   $T,W¤']Z&#65533;¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/10:07:29|
Found 0 errors checking kernel image
20/10:07:30|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/10:07:30|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/10:07:31|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/10:07:31|SPIconfig done now to set up the LMK3200 registers
20/10:07:31|LMK #0 : regInit[0]=0x80000000
20/10:07:31|LMK #0 : regInit[1]=0x10070600
20/10:07:31|LMK #0 : regInit[2]=0x60601
20/10:07:31|LMK #0 : regInit[3]=0x60602
20/10:07:31|LMK #0 : regInit[4]=0x60603
20/10:07:31|LMK #0 : regInit[5]=0x70624
20/10:07:31|LMK #0 : regInit[6]=0x70605
20/10:07:31|LMK #0 : regInit[7]=0x70606
20/10:07:31|LMK #0 : regInit[8]=0x70627
20/10:07:31|LMK #0 : regInit[9]=0x10000908
20/10:07:31|LMK #0 : regInit[10]=0xA0022A09
20/10:07:31|LMK #0 : regInit[11]=0x82800B
20/10:07:31|LMK #0 : regInit[12]=0x28C800D
20/10:07:31|LMK #0 : regInit[13]=0x830020E
20/10:07:31|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/10:07:32|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/10:07:32|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/10:07:32|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/10:07:32|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/10:07:35|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd7 0xcc
20/10:07:42|Using Xilinx Virtex440 machine description
20/10:07:42|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/10:07:42|Zone PFN ranges:
20/10:07:42|  DMA      0x00000000 -> 0x00007000
20/10:07:42|  Normal   0x00007000 -> 0x00007000
20/10:07:42|Movable zone start PFN for each node
20/10:07:42|early_node_map[1] active PFN ranges
20/10:07:43|    0: 0x00000000 -> 0x00007000
20/10:07:43|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/10:07:43|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/10:07:43|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/10:07:43|PID hash table entries: 512 (order: 9, 2048 bytes)
20/10:07:43|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/10:07:43|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/10:07:43|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/10:07:43|Kernel virtual memory layout:
20/10:07:43|  * 0xffffe000..0xfffff000  : fixmap
20/10:07:43|  * 0xfde00000..0xfe000000  : consistent mem
20/10:07:43|  * 0xfde00000..0xfde00000  : early ioremap
20/10:07:43|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/10:07:43|NR_IRQS:512
20/10:07:43|clocksource: timebase mult[a00000] shift[22] registered
20/10:07:43|Console: colour dummy device 80x25
20/10:07:43|Mount-cache hash table entries: 512
20/10:07:43|NET: Registered protocol family 16
20/10:07:43|PCI: Probing PCI hardware
20/10:07:43|bio: create slab <bio-0> at 0
20/10:07:44|NET: Registered protocol family 2
20/10:07:44|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/10:07:44|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/10:07:44|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/10:07:44|TCP: Hash tables configured (established 4096 bind 4096)
20/10:07:44|TCP reno registered
20/10:07:44|NET: Registered protocol family 1
20/10:07:44|ROMFS MTD (C) 2007 Red Hat, Inc.
20/10:07:44|msgmni has been set to 214
20/10:07:44|io scheduler noop registered
20/10:07:44|io scheduler anticipatory registered
20/10:07:44|io scheduler deadline registered
20/10:07:44|io scheduler cfq registered (default)
20/10:07:44|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/10:07:44|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/10:07:44|console [ttyS0] enabled
20/10:07:44|brd: module loaded
20/10:07:44|loop: module loaded
20/10:07:44|Device Tree Probing 'ethernet'
20/10:07:44|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d7:cc
20/10:07:44|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/10:07:44|XLlTemac: DCR address: 0x80
20/10:07:44|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/10:07:45|XLlTemac: Allocating DMA descriptors with kmalloc
20/10:07:45|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/10:07:45|XTemac: PHY detected at address 7.
20/10:07:45|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/10:07:45|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45|Using buffer write method
20/10:07:45|cfi_cmdset_0001: Erase suspend on write enabled
20/10:07:45|cmdlinepart partition parsing not available
20/10:07:45|RedBoot partition parsing not available
20/10:07:45|Creating 5 MTD partitions on "fc000000.flash":
20/10:07:45|0x000000000000-0x000000500000 : "golden_firmware"
20/10:07:45|0x000000500000-0x000000800000 : "golden_kernel"
20/10:07:45|0x000000800000-0x000000d00000 : "user_firmware"
20/10:07:46|0x000000d00000-0x000000fe0000 : "user_kernel"
20/10:07:46|0x000000fe0000-0x000001000000 : "env_variables"
20/10:07:46|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/10:07:46|SPI: XIlinx spi: bus number now 32766
20/10:07:46|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/10:07:46|SPI: XIlinx spi: bus number now 32765
20/10:07:46|mice: PS/2 mouse device common for all mice
20/10:07:46|Device Tree Probing 'i2c'
20/10:07:46| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/10:07:46|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/10:07:46|TCP cubic registered
20/10:07:46|NET: Registered protocol family 17
20/10:07:46|RPC: Registered udp transport module.
20/10:07:46|RPC: Registered tcp transport module.
20/10:07:46|eth0: XLlTemac: Options: 0x3fa
20/10:07:47|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/10:07:47|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/10:07:47|eth0: XLlTemac: speed set to 1000Mb/s
20/10:07:49|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/10:07:49|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/10:07:49|Sending DHCP requests ., OK
20/10:07:50|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.5
20/10:07:50|IP-Config: Complete:
20/10:07:50|     device=eth0, addr=192.168.11.5, mask=255.255.255.0, gw=255.255.255.255,
20/10:07:50|     host=aida05, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/10:07:50|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05
20/10:07:50|Looking up port of RPC 100003/2 on 192.168.11.99
20/10:07:51|Looking up port of RPC 100005/1 on 192.168.11.99
20/10:07:51|VFS: Mounted root (nfs filesystem) on device 0:12.
20/10:07:51|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/10:07:51|Starting sysinit...
20/10:07:51|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/10:07:52|            Press 'I' to enter interactive startup.
20/10:07:52|Setting clock  (utc): Tue Apr 20 10:07:52 BST 2021 [  OK  ]
20/10:07:53|Building the cache [  OK  ]
20/10:07:53|Setting hostname aida05:  [  OK  ]
20/10:07:54|Mounting local filesystems:  [  OK  ]
20/10:07:55|Enabling /etc/fstab swaps:  [  OK  ]
20/10:07:57|Finishing sysinit...
INIT: Entering runlevel: 3
20/10:08:00|Entering non-interactive startup
20/10:08:01|FATAL: Module ipv6 not found.
20/10:08:02|Bringing up loopback interface:  [  OK  ]
20/10:08:04|FATAL: Module ipv6 not found.
20/10:08:04|Starting system logger: [  OK  ]
20/10:08:05|Starting kernel logger: [  OK  ]
20/10:08:05|Starting rpcbind: [  OK  ]
20/10:08:06|Mounting NFS filesystems:  [  OK  ]
20/10:08:07|Mounting other filesystems:  [  OK  ]
20/10:08:07|Starting xinetd: [  OK  ]
20/10:08:08|Starting midas:  Starting MIDAS Data Acquisition for aida05
20/10:08:08|xaida: device parameters: base=0x81000000 size=0x200000
20/10:08:14|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/10:08:14|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/10:08:14|xaida: driver assigned major number 254
20/10:08:14|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/10:08:20|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/10:08:20|AIDAMEM: aidamem: driver assigned major number 253
20/10:08:20|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/10:08:26|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/10:08:27|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/10:08:27|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/10:08:27|Computer Name = aida05; Temp Directory = /tmp/tcl356
20/10:08:30|package limit is not available: can't find package limit
20/10:08:33|Running with default file descriptor limit
20/10:08:33|package setuid is not available: can't find package setuid
20/10:08:35|Running as user 0 group 0
20/10:08:35|
20/10:08:36|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  279   Sat Apr 24 15:13:33 2021 TDaida09 crash - 08:10:52 Tuesday 20 April
From ttyUSB6 system console log

20/07:07:10|kernel BUG at mm/slab.c:2974!
20/07:07:10|Oops: Exception in kernel mode, sig: 5 [#1]
20/07:07:10|PREEMPT Xilinx Virtex440
20/07:07:10|Modules linked in: aidamem xdriver xh_spidev_register
20/07:07:10|NIP: c009211c LR: c00920b0 CTR: c0038e70
20/07:07:10|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)
20/07:07:10|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000
20/07:07:10|TASK = c036e318[0] 'swapper' THREAD: c0390000
20/07:07:10|GPR00: 00000001 c0391ca0 c036e318 c680daf0 00000020 00000001 c6940000 00000000
20/07:07:11|GPR08: fffffff4 00010003 c680e400 c0390000 24008042 00005aa8 c03a0000 00000020
20/07:07:11|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200
20/07:07:11|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400
20/07:07:11|NIP [c009211c] cache_alloc_refill+0x130/0x608
20/07:07:11|LR [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:11|Call Trace:
20/07:07:11|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
20/07:07:11|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:11|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:11|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:11|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:11|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:11|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:11|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:11|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:11|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:11|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:11|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:12|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:12|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:12|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:12|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:12|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:12|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:12|Instruction dump:
20/07:07:12|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010
20/07:07:12|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840
20/07:07:12|Kernel panic - not syncing: Fatal exception in interrupt
20/07:07:12|Call Trace:
20/07:07:12|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/07:07:12|[c0391a80] [c00345bc] panic+0x94/0x168
20/07:07:12|[c0391ad0] [c000bd44] die+0x178/0x18c
20/07:07:12|[c0391af0] [c000c000] _exception+0x164/0x1b4
20/07:07:12|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c
20/07:07:12|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:12|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:12|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:12|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:13|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:13|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:13|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:13|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:13|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:13|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:13|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:13|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:13|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:13|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:13|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:13|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:13|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:13|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:13|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/07:10:14|
Found 0 errors checking kernel image
20/07:10:15|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/07:10:15|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/07:10:15|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/07:10:15|SPIconfig done now to set up the LMK3200 registers
20/07:10:15|LMK #0 : regInit[0]=0x80000000
20/07:10:15|LMK #0 : regInit[1]=0x10070600
20/07:10:15|LMK #0 : regInit[2]=0x60601
20/07:10:15|LMK #0 : regInit[3]=0x60602
20/07:10:16|LMK #0 : regInit[4]=0x60603
20/07:10:16|LMK #0 : regInit[5]=0x70624
20/07:10:16|LMK #0 : regInit[6]=0x70605
20/07:10:16|LMK #0 : regInit[7]=0x70606
20/07:10:16|LMK #0 : regInit[8]=0x70627
20/07:10:16|LMK #0 : regInit[9]=0x10000908
20/07:10:16|LMK #0 : regInit[10]=0xA0022A09
20/07:10:16|LMK #0 : regInit[11]=0x82800B
20/07:10:16|LMK #0 : regInit[12]=0x28C800D
20/07:10:16|LMK #0 : regInit[13]=0x830020E
20/07:10:16|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/07:10:16|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/07:10:16|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/07:10:16|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/07:10:17|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/07:10:20|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xee
20/07:10:26|Using Xilinx Virtex440 machine description
20/07:10:27|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/07:10:27|Zone PFN ranges:
20/07:10:27|  DMA      0x00000000 -> 0x00007000
20/07:10:27|  Normal   0x00007000 -> 0x00007000
20/07:10:27|Movable zone start PFN for each node
20/07:10:27|early_node_map[1] active PFN ranges
20/07:10:27|    0: 0x00000000 -> 0x00007000
20/07:10:27|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/07:10:27|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/07:10:27|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/07:10:27|PID hash table entries: 512 (order: 9, 2048 bytes)
20/07:10:27|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/07:10:27|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/07:10:28|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/07:10:28|Kernel virtual memory layout:
20/07:10:28|  * 0xffffe000..0xfffff000  : fixmap
20/07:10:28|  * 0xfde00000..0xfe000000  : consistent mem
20/07:10:28|  * 0xfde00000..0xfde00000  : early ioremap
20/07:10:28|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/07:10:28|NR_IRQS:512
20/07:10:28|clocksource: timebase mult[a00000] shift[22] registered
20/07:10:28|Console: colour dummy device 80x25
20/07:10:28|Mount-cache hash table entries: 512
20/07:10:28|NET: Registered protocol family 16
20/07:10:28|PCI: Probing PCI hardware
20/07:10:28|bio: create slab <bio-0> at 0
20/07:10:28|NET: Registered protocol family 2
20/07:10:28|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/07:10:28|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/07:10:28|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/07:10:28|TCP: Hash tables configured (established 4096 bind 4096)
20/07:10:28|TCP reno registered
20/07:10:28|NET: Registered protocol family 1
20/07:10:28|ROMFS MTD (C) 2007 Red Hat, Inc.
20/07:10:28|msgmni has been set to 214
20/07:10:29|io scheduler noop registered
20/07:10:29|io scheduler anticipatory registered
20/07:10:29|io scheduler deadline registered
20/07:10:29|io scheduler cfq registered (default)
20/07:10:29|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/07:10:29|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/07:10:29|console [ttyS0] enabled
20/07:10:29|brd: module loaded
20/07:10:29|loop: module loaded
20/07:10:29|Device Tree Probing 'ethernet'
20/07:10:29|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ee
20/07:10:29|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/07:10:29|XLlTemac: DCR address: 0x80
20/07:10:29|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/07:10:29|XLlTemac: Allocating DMA descriptors with kmalloc
20/07:10:29|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/07:10:29|XTemac: PHY detected at address 7.
20/07:10:29|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/07:10:29|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/07:10:29| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30|Using buffer write method
20/07:10:30|cfi_cmdset_0001: Erase suspend on write enabled
20/07:10:30|cmdlinepart partition parsing not available
20/07:10:30|RedBoot partition parsing not available
20/07:10:30|Creating 5 MTD partitions on "fc000000.flash":
20/07:10:30|0x000000000000-0x000000500000 : "golden_firmware"
20/07:10:30|0x000000500000-0x000000800000 : "golden_kernel"
20/07:10:30|0x000000800000-0x000000d00000 : "user_firmware"
20/07:10:30|0x000000d00000-0x000000fe0000 : "user_kernel"
20/07:10:30|0x000000fe0000-0x000001000000 : "env_variables"
20/07:10:30|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/07:10:30|SPI: XIlinx spi: bus number now 32766
20/07:10:30|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/07:10:30|SPI: XIlinx spi: bus number now 32765
20/07:10:30|mice: PS/2 mouse device common for all mice
20/07:10:31|Device Tree Probing 'i2c'
20/07:10:31| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/07:10:31|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/07:10:31|TCP cubic registered
20/07:10:31|NET: Registered protocol family 17
20/07:10:31|RPC: Registered udp transport module.
20/07:10:31|RPC: Registered tcp transport module.
20/07:10:31|eth0: XLlTemac: Options: 0x3fa
20/07:10:31|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/07:10:31|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/07:10:31|eth0: XLlTemac: speed set to 1000Mb/s
20/07:10:33|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/07:10:34|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/07:10:34|Sending DHCP requests ., OK
20/07:10:35|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
20/07:10:35|IP-Config: Complete:
20/07:10:35|     device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
20/07:10:35|     host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/07:10:35|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
20/07:10:35|Looking up port of RPC 100003/2 on 192.168.11.99
20/07:10:35|Looking up port of RPC 100005/1 on 192.168.11.99
20/07:10:35|VFS: Mounted root (nfs filesystem) on device 0:12.
20/07:10:35|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/07:10:36|Starting sysinit...
20/07:10:36|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/07:10:37|            Press 'I' to enter interactive startup.
20/07:10:37|Setting clock  (utc): Tue Apr 20 07:10:37 BST 2021 [  OK  ]
20/07:10:38|Building the cache [  OK  ]
20/07:10:38|Setting hostname aida09:  [  OK  ]
20/07:10:39|Mounting local filesystems:  [  OK  ]
20/07:10:40|Enabling /etc/fstab swaps:  [  OK  ]
20/07:10:43|Finishing sysinit...
INIT: Entering runlevel: 3
20/07:10:46|Entering non-interactive startup
20/07:10:46|FATAL: Module ipv6 not found.
20/07:10:47|Bringing up loopback interface:  [  OK  ]
20/07:10:49|FATAL: Module ipv6 not found.
20/07:10:50|Starting system logger: [  OK  ]
20/07:10:50|Starting kernel logger: [  OK  ]
20/07:10:50|Starting rpcbind: [  OK  ]
20/07:10:52|Mounting NFS filesystems:  [  OK  ]
20/07:10:53|Mounting other filesystems:  [  OK  ]
20/07:10:53|Starting xinetd: [  OK  ]
20/07:10:54|Starting midas:  Starting MIDAS Data Acquisition for aida09
20/07:10:54|xaida: device parameters: base=0x81000000 size=0x200000
20/07:11:00|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/07:11:00|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/07:11:00|xaida: driver assigned major number 254
20/07:11:00|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/07:11:06|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/07:11:06|AIDAMEM: aidamem: driver assigned major number 253
20/07:11:06|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/07:11:12|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/07:11:12|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/07:11:12|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/07:11:13|Computer Name = aida09; Temp Directory = /tmp/tcl361
20/07:11:17|package limit is not available: can't find package limit
20/07:11:20|Running with default file descriptor limit
20/07:11:20|package setuid is not available: can't find package setuid
20/07:11:21|Running as user 0 group 0
20/07:11:21|
20/07:11:22|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
20/07:11:22|
20/07:11:22|Built without pthreads
20/07:11:22|xaida: open:
20/07:11:22|
20/07:11:22|Creating NetAIDAMEM: aidamem_open:
20/07:11:22|Vars
20/07:11:22|Output buffer length = 65504; format option = 4; transfer option = 3
20/07:11:22|EB transfer option = 3
20/07:11:22|NetVars created and initialised
20/07:11:22|Statistics thread starting
20/07:11:22|Statistics thread created
20/07:11:22|Stat/Rate creation thread starting
20/07:11:22|Data Acquisition task has PID 375
20/07:11:22|Stat/Rate creation thread created
20/07:11:22|Hit/Rate creation thread starting
20/07:11:22|Hit/Rate creation thread created
20/07:11:22|AIDA Heartbeat thread starting
20/07:11:22|Heartbeat thread created
20/07:11:22|Installing signal handlers
20/07:11:22|Done
20/07:11:22|ModuleNum = 0
20/07:11:22|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706
20/07:11:22|Spectra table initialised
20/07:11:22|AIDA Data Acquisition now all ready to start
20/07:11:23|SIGBUS, SIGSEGV and SIGPIPE traps setup
20/07:11:23|/debug user "debug" password "5tvq-4iyxl 4"
20/07:11:30|httpd started on port 8015
20/07:11:30|
20/07:11:30|Cannot use /MIDAS/config/TclHttpd/aida09@8015/startup.tcl
20/07:11:30|Custom startup from /MIDAS/config/TclHttpd/aida09/startup.tcl
20/07:11:30|XAIDA Access package 1.0
20/07:11:32|/XAIDAAccessServer
20/07:11:32|XAD9252 Access package 1.0
20/07:11:32|/XAD9252AccessServer
20/07:11:32|/DataBaseAccessServer
20/07:11:32|[  OK  ]
20/07:11:32|/NetVarService
20/07:11:32|/SigTaskService
20/07:11:32|Loaded MemSasAccess
20/07:11:33|/SpectrumService
20/07:11:33|loading tcl/AIDARunControl.tcl for namespace ::
20/07:11:33|/DataAcquisitionControlServer
20/07:11:33|DefineMessage unknown
20/07:11:33|
20/07:11:33|Run Control Server Implementation for AIDA
20/07:11:33|DENX ELDK version 4.2 build 2008-04-01
20/07:11:33|Linux 2.6.31 on a ppc
20/07:11:33|
20/07:11:33|aida09 login: RunControlServer loaded
20/07:11:33|loading Html/RunControl/implementation.tcl
20/07:11:33|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
20/07:11:34|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
20/07:11:34|Created UI registers
20/07:11:34|RunControl loaded
20/07:11:34|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
20/07:11:34|AIDA RunControl loaded
20/07:11:35|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
20/07:11:35|do_GetState returned z=0 and 8
20/07:12:03|do_GetState returned z=0 and 8
20/07:12:09|do_GetState returned z=0 and 8
20/07:12:21|xaida: open:
20/07:12:35|do_GetState returned z=0 and 8
20/07:12:36|
  278   Sat Apr 24 13:33:44 2021 TDaida01 restart - 22.07UTC+2 Tuesday 20 April
From aida01 /var/log/messages

Apr 20 21:04:06 aida01 kernel: BUG: scheduling while atomic: tclsh8.5/398/0x00000201
Apr 20 21:04:08 aida01 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 20 21:04:09 aida01 kernel: Call Trace:
Apr 20 21:04:11 aida01 kernel: [c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
Apr 20 21:04:12 aida01 kernel: [c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
Apr 20 21:04:14 aida01 kernel: [c6941aa0] [c02a8020] schedule+0x2f4/0x34c
Apr 20 21:07:53 aida01 syslogd 1.4.2: restart.

From ttyUSB0 system console log

20/21:04:06|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:06|Call Trace:
20/21:04:06|[c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:06|[c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:07|[c6941aa0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:07|[c6941ae0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:07|[c6941af0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:07|[c6941b10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:07|[c6941b50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:07|[c6941b90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:07|[c6941ba0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:07|[c6941bd0] [c013da58] nfs3_rpc_wrapper+0x48/0x84
20/21:04:07|[c6941bf0] [c013e734] nfs3_proc_access+0xcc/0x1ac
20/21:04:07|[c6941cc0] [c0129644] nfs_do_access+0xb8/0x438
20/21:04:07|[c6941d00] [c0129a6c] nfs_permission+0xa8/0x1c4
20/21:04:07|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:07|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:07|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:07|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:07|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:07|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:07|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:07|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:07|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:08|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:08|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:08|Call Trace:
20/21:04:08|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:08|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:08|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:08|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:08|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:08|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:08|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:08|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:08|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:08|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:08|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:08|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:08|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:08|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:09|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:09|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:09|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:09|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:09|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:09|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:09|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:09|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:09|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:09|Call Trace:
20/21:04:09|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:09|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:09|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:09|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:09|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:09|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:09|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:09|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:09|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:09|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:10|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:10|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:10|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:10|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:10|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:10|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:10|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:10|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:10|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:10|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:10|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:10|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:10|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:10|Call Trace:
20/21:04:10|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:10|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:10|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:10|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:10|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:10|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:11|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:11|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:11|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:11|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:11|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:11|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:11|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:11|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:11|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:11|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:11|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:11|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:11|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:11|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:11|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:11|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:11|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:11|Call Trace:
20/21:04:11|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:11|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:12|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:12|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:12|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:12|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:12|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:12|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:12|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:12|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:12|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:12|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:12|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:12|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:12|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:12|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:12|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:12|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:12|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:12|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:12|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:12|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:13|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:13|Call Trace:
20/21:04:13|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:13|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:13|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:13|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:13|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:13|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:13|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:13|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:13|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:13|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:13|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:13|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:13|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:13|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:13|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:13|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:13|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:13|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:14|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:14|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:14|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:14|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:14|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:14|Call Trace:
20/21:04:14|[c6941950] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:14|[c6941990] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:14|[c69419a0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:14|[c69419e0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:14|[c69419f0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:14|[c6941a10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:14|[c6941a50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:14|[c6941a90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:14|[c6941aa0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:14|[c6941ad0] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:14|[c6941b00] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:14|[c6941bc0] [c012bd70] nfs_lookup_revalidate+0x3c0/0x464
20/21:04:14|[c6941d00] [c00a0f7c] do_lookup+0x54/0x20c
20/21:04:14|[c6941d30] [c00a308c] __link_path_walk+0x57c/0xf7c
20/21:04:15|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:15|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:15|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:15|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:15|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:15|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:15|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:15|Page fault in user mode with in_atomic() = 1 mm = c6178c80
20/21:04:15|NIP = fb85a30  MSR = 2d000
20/21:04:15|Oops: Weird page fault, sig: 11 [#1]
20/21:04:15|PREEMPT Xilinx Virtex440
20/21:04:15|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:15|NIP: 0fb85a30 LR: 0ff85338 CTR: c01288f4
20/21:04:15|REGS: c6941f50 TRAP: 0401   Not tainted  (2.6.31)
20/21:04:15|MSR: 0002d000 <EE,PR,ME,CE>  CR: 44884044  XER: 00000000
20/21:04:15|TASK = c608e0c0[398] 'tclsh8.5' THREAD: c6940000
20/21:04:15|GPR00: 0ff85334 bf9a3870 4802b0a0 ffffffff 0fc094e0 54884044 fffffff2 0fc08d3c
20/21:04:15|GPR08: 0002d000 480240a8 00000000 bf9a3870 c6940000
20/21:04:15|NIP [0fb85a30] 0xfb85a30
20/21:04:15|LR [0ff85338] 0xff85338
20/21:04:16|Call Trace:
20/21:04:16|Kernel panic - not syncing: Fatal exception in interrupt
20/21:04:16|Call Trace:
20/21:04:16|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:16|[c6941e20] [c00345bc] panic+0x94/0x168
20/21:04:16|[c6941e70] [c000bd44] die+0x178/0x18c
20/21:04:16|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
20/21:04:16|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
20/21:04:16|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/21:07:16|
Found 0 errors checking kernel image
20/21:07:18|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/21:07:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/21:07:18|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/21:07:18|SPIconfig done now to set up the LMK3200 registers
20/21:07:18|LMK #0 : regInit[0]=0x80000000
20/21:07:18|LMK #0 : regInit[1]=0x10070600
20/21:07:18|LMK #0 : regInit[2]=0x60601
20/21:07:18|LMK #0 : regInit[3]=0x60602
20/21:07:18|LMK #0 : regInit[4]=0x60603
20/21:07:18|LMK #0 : regInit[5]=0x70624
20/21:07:18|LMK #0 : regInit[6]=0x70605
20/21:07:18|LMK #0 : regInit[7]=0x70606
20/21:07:18|LMK #0 : regInit[8]=0x70627
20/21:07:18|LMK #0 : regInit[9]=0x10000908
20/21:07:18|LMK #0 : regInit[10]=0xA0022A09
20/21:07:18|LMK #0 : regInit[11]=0x82800B
20/21:07:18|LMK #0 : regInit[12]=0x28C800D
20/21:07:18|LMK #0 : regInit[13]=0x830020E
20/21:07:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/21:07:19|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/21:07:19|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
ADCs calibrated
20/21:07:19|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
20/21:07:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/21:07:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xba 0x8a
20/21:07:29|Using Xilinx Virtex440 machine description
20/21:07:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/21:07:30|Zone PFN ranges:
20/21:07:30|  DMA      0x00000000 -> 0x00007000
20/21:07:30|  Normal   0x00007000 -> 0x00007000
20/21:07:30|Movable zone start PFN for each node
20/21:07:30|early_node_map[1] active PFN ranges
20/21:07:30|    0: 0x00000000 -> 0x00007000
20/21:07:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/21:07:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/21:07:30|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/21:07:30|PID hash table entries: 512 (order: 9, 2048 bytes)
20/21:07:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/21:07:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/21:07:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/21:07:30|Kernel virtual memory layout:
20/21:07:30|  * 0xffffe000..0xfffff000  : fixmap
20/21:07:31|  * 0xfde00000..0xfe000000  : consistent mem
20/21:07:31|  * 0xfde00000..0xfde00000  : early ioremap
20/21:07:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/21:07:31|NR_IRQS:512
20/21:07:31|clocksource: timebase mult[a00000] shift[22] registered
20/21:07:31|Console: colour dummy device 80x25
20/21:07:31|Mount-cache hash table entries: 512
20/21:07:31|NET: Registered protocol family 16
20/21:07:31|PCI: Probing PCI hardware
20/21:07:31|bio: create slab <bio-0> at 0
20/21:07:31|NET: Registered protocol family 2
20/21:07:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/21:07:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/21:07:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/21:07:31|TCP: Hash tables configured (established 4096 bind 4096)
20/21:07:31|TCP reno registered
20/21:07:31|NET: Registered protocol family 1
20/21:07:31|ROMFS MTD (C) 2007 Red Hat, Inc.
20/21:07:31|msgmni has been set to 214
20/21:07:31|io scheduler noop registered
20/21:07:31|io scheduler anticipatory registered
20/21:07:31|io scheduler deadline registered
20/21:07:31|io scheduler cfq registered (default)
20/21:07:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/21:07:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/21:07:32|console [ttyS0] enabled
20/21:07:32|brd: module loaded
20/21:07:32|loop: module loaded
20/21:07:32|Device Tree Probing 'ethernet'
20/21:07:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ba:8a
20/21:07:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/21:07:32|XLlTemac: DCR address: 0x80
20/21:07:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/21:07:32|XLlTemac: Allocating DMA descriptors with kmalloc
20/21:07:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/21:07:32|XTemac: PHY detected at address 7.
20/21:07:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/21:07:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33|Using buffer write method
20/21:07:33|cfi_cmdset_0001: Erase suspend on write enabled
20/21:07:33|cmdlinepart partition parsing not available
20/21:07:33|RedBoot partition parsing not available
20/21:07:33|Creating 5 MTD partitions on "fc000000.flash":
20/21:07:33|0x000000000000-0x000000500000 : "golden_firmware"
20/21:07:33|0x000000500000-0x000000800000 : "golden_kernel"
20/21:07:33|0x000000800000-0x000000d00000 : "user_firmware"
20/21:07:33|0x000000d00000-0x000000fe0000 : "user_kernel"
20/21:07:33|0x000000fe0000-0x000001000000 : "env_variables"
20/21:07:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/21:07:33|SPI: XIlinx spi: bus number now 32766
20/21:07:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/21:07:33|SPI: XIlinx spi: bus number now 32765
20/21:07:33|mice: PS/2 mouse device common for all mice
20/21:07:33|Device Tree Probing 'i2c'
20/21:07:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/21:07:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/21:07:33|TCP cubic registered
20/21:07:33|NET: Registered protocol family 17
20/21:07:34|RPC: Registered udp transport module.
20/21:07:34|RPC: Registered tcp transport module.
20/21:07:34|eth0: XLlTemac: Options: 0x3fa
20/21:07:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/21:07:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/21:07:34|eth0: XLlTemac: speed set to 1000Mb/s
20/21:07:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/21:07:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/21:07:36|Sending DHCP requests ., OK
20/21:07:38|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.1
20/21:07:38|IP-Config: Complete:
20/21:07:38|     device=eth0, addr=192.168.11.1, mask=255.255.255.0, gw=255.255.255.255,
20/21:07:38|     host=aida01, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/21:07:38|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01
20/21:07:38|Looking up port of RPC 100003/2 on 192.168.11.99
20/21:07:38|Looking up port of RPC 100005/1 on 192.168.11.99
20/21:07:38|VFS: Mounted root (nfs filesystem) on device 0:12.
20/21:07:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/21:07:39|Starting sysinit...
20/21:07:39|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/21:07:40|            Press 'I' to enter interactive startup.
20/21:07:40|Setting clock  (utc): Tue Apr 20 21:07:40 BST 2021 [  OK  ]
20/21:07:40|Building the cache [  OK  ]
20/21:07:40|Setting hostname aida01:  [  OK  ]
20/21:07:42|Mounting local filesystems:  [  OK  ]
20/21:07:43|Enabling /etc/fstab swaps:  [  OK  ]
20/21:07:46|Finishing sysinit...
INIT: Entering runlevel: 3
20/21:07:49|Entering non-interactive startup
20/21:07:49|FATAL: Module ipv6 not found.
20/21:07:51|Bringing up loopback interface:  [  OK  ]
20/21:07:53|FATAL: Module ipv6 not found.
20/21:07:53|Starting system logger: [  OK  ]
20/21:07:54|Starting kernel logger: [  OK  ]
20/21:07:54|Starting rpcbind: [  OK  ]
20/21:07:56|Mounting NFS filesystems:  [  OK  ]
20/21:07:57|Mounting other filesystems:  [  OK  ]
20/21:07:57|Starting xinetd: [  OK  ]
20/21:07:58|Starting midas:  Starting MIDAS Data Acquisition for aida01
20/21:07:58|xaida: device parameters: base=0x81000000 size=0x200000
20/21:08:04|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/21:08:04|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/21:08:05|xaida: driver assigned major number 254
20/21:08:05|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/21:08:10|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/21:08:10|AIDAMEM: aidamem: driver assigned major number 253
20/21:08:10|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/21:08:17|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/21:08:17|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/21:08:17|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/21:08:17|Computer Name = aida01; Temp Directory = /tmp/tcl361
20/21:08:22|package limit is not available: can't find package limit
20/21:08:25|Running with default file descriptor limit
20/21:08:25|
20/21:08:26|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  277   Sat Apr 24 13:25:23 2021 NH, TDAIDA FEE64 crash log for S460
https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0
  276   Sat Apr 24 10:21:06 2021 NHSaturday 24.04.2021
I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again

Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)

System Wide Checks
Clocks: OK
FADC: OK
WR: 
		 Base 		Current 	Difference
aida05 fault 	 0x31a2 : 	 0x31a2 : 	 0  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22e8 : 	 1  
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK

aida05 ASIC maybe didn't complete setup properly (WR 0x10?)

Timestamps: OK (1)
Temps: OK

Will start gzipping S460 runs for migration to lustre & tape
Attachment 1: AnyDesk_2021-04-24_11-27-49.png
AnyDesk_2021-04-24_11-27-49.png
Attachment 2: AnyDesk_2021-04-24_11-28-50.png
AnyDesk_2021-04-24_11-28-50.png
  275   Sat Apr 24 05:17:51 2021 MLEnd of beam

Beam has now stopped (end of expriment) and background measurements starts.

  274   Sat Apr 24 04:50:01 2021 MLSpectra

Some screenshot of spectra attached

Attachment 1: 24-04-2021_540am_SpecLayoutId3.png
24-04-2021_540am_SpecLayoutId3.png
Attachment 2: 24-04-2021_540am_SpecLayId4.png
24-04-2021_540am_SpecLayId4.png
Attachment 3: 24-04-2021_540am_SpecLayId5.png
24-04-2021_540am_SpecLayId5.png
Attachment 4: 24-04-2021_540am_SpecLayId6.png
24-04-2021_540am_SpecLayId6.png
Attachment 5: 24-04-2021_540am_SpecLayId7.png
24-04-2021_540am_SpecLayId7.png
Attachment 6: 24-04-2021_540am_SpecLayId8.png
24-04-2021_540am_SpecLayId8.png
Attachment 7: 24-04-2021_540am_SpecLayId1.png
24-04-2021_540am_SpecLayId1.png
Attachment 8: 24-04-2021_540am_SpecLayId2.png
24-04-2021_540am_SpecLayId2.png
  273   Sat Apr 24 04:07:23 2021 MLucesb scalers

ucesb is back - see attachment

Attachment 1: 24-04-2021_5am_ucesb.png
24-04-2021_5am_ucesb.png
  272   Sat Apr 24 03:32:11 2021 MLquick checks

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

 

Attachment 1: 24-04-2021_430am_GoodEvent.png
24-04-2021_430am_GoodEvent.png
  271   Sat Apr 24 03:16:19 2021 TDaida-gsi window manager/login session crashed
Looks like the aida-gsi window manager/login session crashed.

On site S460 shift restarted AnyDesk. Upon re-connection *all* browsers, terminals, applications were gone
I have re-started the basics but BNC PB-5 pulser, CAEN NIM bin, MBS rate monitor, system console monitor etc
still need to be re-started. 

Extract from /var/log/messages

Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal cpuset=/ mems_allowed=0
Apr 24 03:19:01 aidas-gsi kernel: Pid: 18955, comm: gnome-terminal Not tainted 2.6.32-696.el6.x86_64 #1
Apr 24 03:19:01 aidas-gsi kernel: Call Trace:
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131550>] ? dump_header+0x90/0x1b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8123c1dc>] ? security_real_capable_noaudit+0x3c/0x70
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811319d2>] ? oom_kill_process+0x82/0x2a0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131911>] ? select_bad_process+0xe1/0x120
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131e10>] ? out_of_memory+0x220/0x3c0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113e7ec>] ? __alloc_pages_nodemask+0x93c/0x950
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81015533>] ? native_sched_clock+0x13/0x80
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff810aaea1>] ? lock_hrtimer_base+0x31/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81177a3a>] ? alloc_pages_current+0xaa/0x110
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113bb4e>] ? __get_free_pages+0xe/0x50
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1944>] ? __pollwait+0xb4/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81348684>] ? n_tty_poll+0x194/0x1d0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff813440aa>] ? tty_poll+0x8a/0xa0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1db5>] ? do_sys_poll+0x2d5/0x5b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1890>] ? __pollwait+0x0/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1745>] ? poll_select_set_timeout+0x95/0xb0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b2281>] ? sys_poll+0x71/0x100
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Apr 24 03:19:01 aidas-gsi kernel: Mem-Info:
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 172
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  45
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd:  97
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 142
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  49
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd:  56
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 179
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd:  67
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 107
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 170
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  54
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 118
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 196
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  36
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  37
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd: 128
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 173
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd: 161
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 177
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 178
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  99
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 183
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 162
Apr 24 03:19:01 aidas-gsi kernel: active_anon:3408671 inactive_anon:401328 isolated_anon:128
Apr 24 03:19:01 aidas-gsi kernel: active_file:22995 inactive_file:61739 isolated_file:0
Apr 24 03:19:01 aidas-gsi kernel: unevictable:0 dirty:59856 writeback:22 unstable:0
Apr 24 03:19:01 aidas-gsi kernel: free:35843 slab_reclaimable:15134 slab_unreclaimable:65491
Apr 24 03:19:01 aidas-gsi kernel: mapped:129529 shmem:249820 pagetables:24586 bounce:0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA free:15752kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 874 16024 16024
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 free:64204kB min:3680kB low:4600kB high:5520kB active_anon:369500kB inactive_anon:377436kB active_file:10988kB 
inactive_file:26828kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:895084kB mlocked:0kB dirty:26856kB writeback:88kB mapped:35896kB shmem:39156kB 
slab_reclaimable:4224kB slab_unreclaimable:4344kB kernel_stack:3344kB pagetables:2032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:65792 all_unreclaimable? 
no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 15150 15150
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal free:63416kB min:63836kB low:79792kB high:95752kB active_anon:13265184kB inactive_anon:1227876kB active_file:80992kB 
inactive_file:220128kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15513600kB mlocked:0kB dirty:212568kB writeback:0kB mapped:482220kB 
shmem:960124kB slab_reclaimable:56312kB slab_unreclaimable:257620kB kernel_stack:17600kB pagetables:96312kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:17824 all_unreclaimable? no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 0 0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32: 373*4kB 559*8kB 428*16kB 296*32kB 189*64kB 111*128kB 43*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 64204kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal: 16100*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64408kB
Apr 24 03:19:01 aidas-gsi kernel: 357788 total pagecache pages
Apr 24 03:19:01 aidas-gsi kernel: 22930 pages in swap cache
Apr 24 03:19:01 aidas-gsi kernel: Swap cache stats: add 3720478, delete 3697548, find 5026852/5228898
Apr 24 03:19:01 aidas-gsi kernel: Free swap  = 1826100kB
Apr 24 03:19:01 aidas-gsi kernel: Total swap = 8200188kB
Apr 24 03:19:01 aidas-gsi kernel: 4194303 pages RAM
Apr 24 03:19:01 aidas-gsi kernel: 97868 pages reserved
Apr 24 03:19:01 aidas-gsi kernel: 578727 pages shared
Apr 24 03:19:01 aidas-gsi kernel: 3846484 pages non-shared
Apr 24 03:19:01 aidas-gsi kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 24 03:19:01 aidas-gsi kernel: [ 1388]     0  1388     2819      130   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 3927]     0  3927     7442      128  12     -17         -1000 auditd
Apr 24 03:19:01 aidas-gsi kernel: [ 3961]     0  3961    62914      755  15       0             0 rsyslogd
Apr 24 03:19:01 aidas-gsi kernel: [ 4029]     0  4029     4604      141   7       0             0 irqbalance
Apr 24 03:19:01 aidas-gsi kernel: [ 4054]    32  4054     4746      124  13       0             0 rpcbind
Apr 24 03:19:01 aidas-gsi kernel: [ 4074]    81  4074     8194      440   6       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 4089]     0  4089    26490     2647   4       0             0 NetworkManager
Apr 24 03:19:01 aidas-gsi kernel: [ 4096]     0  4096    14534      328  12       0             0 modem-manager
Apr 24 03:19:01 aidas-gsi kernel: [ 4112]    29  4112     5839       97   0       0             0 rpc.statd
Apr 24 03:19:01 aidas-gsi kernel: [ 4151]     0  4151    47297      117  15       0             0 cupsd
Apr 24 03:19:01 aidas-gsi kernel: [ 4171]     0  4171    11251       75   9       0             0 wpa_supplicant
Apr 24 03:19:01 aidas-gsi kernel: [ 4180]     0  4180     2280      162   9       0             0 dhclient
Apr 24 03:19:01 aidas-gsi kernel: [ 4210]     0  4210     1020       82   8       0             0 acpid
Apr 24 03:19:01 aidas-gsi kernel: [ 4222]    68  4222     9968      569  15       0             0 hald
Apr 24 03:19:01 aidas-gsi kernel: [ 4223]     0  4223     5100      184  12       0             0 hald-runner
Apr 24 03:19:01 aidas-gsi kernel: [ 4271]     0  4271     5630      157  12       0             0 hald-addon-inpu
Apr 24 03:19:01 aidas-gsi kernel: [ 4272]    68  4272     4502      120   0       0             0 hald-addon-acpi
Apr 24 03:19:01 aidas-gsi kernel: [ 4306]     0  4306    96540      161   0       0             0 automount
Apr 24 03:19:01 aidas-gsi kernel: [ 4492]     0  4492    26831       15   9       0             0 rpc.rquotad
Apr 24 03:19:01 aidas-gsi kernel: [ 4497]     0  4497     5450      213   4       0             0 rpc.mountd
Apr 24 03:19:01 aidas-gsi kernel: [ 4542]     0  4542     5776       42   4       0             0 rpc.idmapd
Apr 24 03:19:01 aidas-gsi kernel: [ 4567]     0  4567    16573      113   3     -17         -1000 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 4578]     0  4578     5429      117   7       0             0 xinetd
Apr 24 03:19:01 aidas-gsi kernel: [ 4589]    38  4589     7686      213  10       0             0 ntpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4675]     0  4675    20260      147   5       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [ 4682]    89  4682    20323      141   5       0             0 qmgr
Apr 24 03:19:01 aidas-gsi kernel: [ 4696]     0  4696    27153      253   0       0             0 ksmtuned
Apr 24 03:19:01 aidas-gsi kernel: [ 4709]     0  4709    29222      137  10       0             0 crond
Apr 24 03:19:01 aidas-gsi kernel: [ 4724]     0  4724     5277       50  12       0             0 atd
Apr 24 03:19:01 aidas-gsi kernel: [ 4741]     0  4741   172796     2328   0       0             0 virt-who
Apr 24 03:19:01 aidas-gsi kernel: [ 4762]     0  4762   256599      407  10       0             0 libvirtd
Apr 24 03:19:01 aidas-gsi kernel: [ 4858]     0  4858    16126      150  10       0             0 certmonger
Apr 24 03:19:01 aidas-gsi kernel: [ 4903]    99  4903     3223      102   2       0             0 dnsmasq
Apr 24 03:19:01 aidas-gsi kernel: [ 4918]     0  4918     4539      223   5       0             0 dhcpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4920]     0  4920    33509      312  11       0             0 gdm-binary
Apr 24 03:19:01 aidas-gsi kernel: [ 4927]     0  4927     1016       61  11       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4929]     0  4929     1016       61  12       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4931]     0  4931     1016       61   9       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4933]     0  4933     1016       61  15       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4935]     0  4935     1016       61   1       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4944]     0  4944     3148      440   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 4945]     0  4945     3148      438   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 5050]     0  5050  1029198      377   3       0             0 console-kit-dae
Apr 24 03:19:01 aidas-gsi kernel: [ 5126]     0  5126    12508      365   3       0             0 devkit-power-da
Apr 24 03:19:01 aidas-gsi kernel: [ 5164]     0  5164    45435      581   1       0             0 polkitd
Apr 24 03:19:01 aidas-gsi kernel: [ 5175]   499  5175    42114      146   0       0             0 rtkit-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5702]     0  5702    14009      514   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5725]     0  5725    11306       74   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18391]     0 18391    41240      293   8       0             0 gdm-simple-slav
Apr 24 03:19:01 aidas-gsi kernel: [18393]     0 18393  3476635  2412883  13       0             0 Xorg
Apr 24 03:19:01 aidas-gsi kernel: [18427]    42 18427     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18464]     0 18464    47926      269   0       0             0 gdm-session-wor
Apr 24 03:19:01 aidas-gsi kernel: [18510]   500 18510    57281      424   1       0             0 gnome-keyring-d
Apr 24 03:19:01 aidas-gsi kernel: [18520]   500 18520    64493      751   0       0             0 gnome-session
Apr 24 03:19:01 aidas-gsi kernel: [18528]   500 18528     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18529]   500 18529     8211      379   8       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18604]   500 18604    33379     1113  13       0             0 gconfd-2
Apr 24 03:19:01 aidas-gsi kernel: [18612]   500 18612   139831     1984   9       0             0 gnome-settings-
Apr 24 03:19:01 aidas-gsi kernel: [18614]   500 18614    71643      696  12       0             0 seahorse-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18616]   500 18616    34351      298  13       0             0 gvfsd
Apr 24 03:19:01 aidas-gsi kernel: [18636]   500 18636   112427     2321   4       0             0 metacity
Apr 24 03:19:01 aidas-gsi kernel: [18637]   500 18637   203008     6568  10       0             0 gnome-panel
Apr 24 03:19:01 aidas-gsi kernel: [18638]   500 18638   326834    16727  10       0             0 nautilus
Apr 24 03:19:01 aidas-gsi kernel: [18640]   500 18640   141364      341  14       0             0 bonobo-activati
Apr 24 03:19:01 aidas-gsi kernel: [18642]   500 18642    65558      741  10       0             0 bluetooth-apple
Apr 24 03:19:01 aidas-gsi kernel: [18645]   500 18645    68687      968   4       0             0 gnome-power-man
Apr 24 03:19:01 aidas-gsi kernel: [18646]   500 18646     7601      812  12       0             0 restorecond
Apr 24 03:19:01 aidas-gsi kernel: [18648]   500 18648    28879      401   5       0             0 im-settings-dae
Apr 24 03:19:01 aidas-gsi kernel: [18649]   500 18649   132278     1323   4       0             0 gnome-volume-co
Apr 24 03:19:01 aidas-gsi kernel: [18654]   500 18654    75318     2282  10       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18657]   500 18657   119660     1244   2       0             0 trashapplet
Apr 24 03:19:01 aidas-gsi kernel: [18659]   500 18659    88087     3906   5       0             0 wnck-applet
Apr 24 03:19:01 aidas-gsi kernel: [18660]   500 18660    62988      653   7       0             0 vino-server
Apr 24 03:19:01 aidas-gsi kernel: [18661]   500 18661    72490      958   0       0             0 polkit-gnome-au
Apr 24 03:19:01 aidas-gsi kernel: [18662]   500 18662    82733     1441   5       0             0 nm-applet
Apr 24 03:19:01 aidas-gsi kernel: [18666]   500 18666   126400      635   1       0             0 gvfs-gdu-volume
Apr 24 03:19:01 aidas-gsi kernel: [18667]   500 18667    74660     1042   5       0             0 gpk-update-icon
Apr 24 03:19:01 aidas-gsi kernel: [18669]   500 18669    64033      850   0       0             0 gdu-notificatio
Apr 24 03:19:01 aidas-gsi kernel: [18676]   500 18676    81107     2344   6       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [18693]   500 18693   131702     2471   3       0             0 pulseaudio
Apr 24 03:19:01 aidas-gsi kernel: [18823]   500 18823    58170      269   4       0             0 gvfs-afc-volume
Apr 24 03:19:01 aidas-gsi kernel: [18836]   500 18836    37789      300   8       0             0 gvfs-gphoto2-vo
Apr 24 03:19:01 aidas-gsi kernel: [18844]   500 18844    36051      437   0       0             0 gvfsd-trash
Apr 24 03:19:01 aidas-gsi kernel: [18868]   500 18868    10275      287   6       0             0 gconf-im-settin
Apr 24 03:19:01 aidas-gsi kernel: [18869]   500 18869    84055     4872   3       0             0 gnome-screensav
Apr 24 03:19:01 aidas-gsi kernel: [18870]   500 18870     2057       88   3       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18871]   500 18871    27123      198   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18894]   500 18894    98681     1288   0       0             0 gdm-user-switch
Apr 24 03:19:01 aidas-gsi kernel: [18896]   500 18896    72800      911   3       0             0 notification-ar
Apr 24 03:19:01 aidas-gsi kernel: [18897]   500 18897   143660     1846   2       0             0 clock-applet
Apr 24 03:19:01 aidas-gsi kernel: [18900]   500 18900    23718      242   3       0             0 gconf-helper
Apr 24 03:19:01 aidas-gsi kernel: [18918]   500 18918    34358      265   0       0             0 gvfsd-burn
Apr 24 03:19:01 aidas-gsi kernel: [18932]   500 18932    33897      307   4       0             0 gvfsd-metadata
Apr 24 03:19:01 aidas-gsi kernel: [18955]   500 18955    82196     7037   7       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18957]   500 18957     2057       97   7       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18977]   500 18977    27123      114   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18991]     0 18991    47361      130  10       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [18992]     0 18992    27524      290  15       0             0 iptraf
Apr 24 03:19:01 aidas-gsi kernel: [21973]   500 21973  1058234   192783   0       0             0 firefox
Apr 24 03:19:01 aidas-gsi kernel: [21987]   500 21987    68608      284   3       0             0 at-spi-bus-laun
Apr 24 03:19:01 aidas-gsi kernel: [22221]   500 22221   731047    34381   0       0             0 Privileged Cont
Apr 24 03:19:01 aidas-gsi kernel: [22248]   500 22248   705574    10970   0       0             0 WebExtensions
Apr 24 03:19:01 aidas-gsi kernel: [22335]   500 22335   960920   166074   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22407]   500 22407  1023120    77418   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22810]   500 22810    27123      114   2       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [22826]   500 22826    15487      185   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [22830]   500 22830    27123      114   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26234]   500 26234   839272    43450   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [26311]   500 26311    27123      282   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26884]   500 26884    27123      115   5       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19388]   500 19388    27123      293   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11683]   500 11683   791783    18526   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [ 4876]   500  4876    54985      304   5       0             0 gvfsd-computer
Apr 24 03:19:01 aidas-gsi kernel: [17248]   500 17248    29595      100   1       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17249]   500 17249    29631      157  10       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17250]   500 17250    27124      114   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17266]   500 17266    15489      181   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [19879]     0 19879    25531      154   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19883]   500 19883    25531      163   2       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19884]   500 19884    27638      106  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19907]   500 19907    29595       97   5       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [14850]   500 14850    56156      327   2       0             0 gvfsd-network
Apr 24 03:19:01 aidas-gsi kernel: [14855]   500 14855    36479      318   3       0             0 gvfsd-dnssd
Apr 24 03:19:01 aidas-gsi kernel: [22615]   500 22615    27123      117   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [13867]     0 13867    25531      161   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13880]   500 13880    25645      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13881]   500 13881    14400      138   8       0             0 sftp-server
Apr 24 03:19:01 aidas-gsi kernel: [20258]   500 20258   776888    93602   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [25910]   500 25910    15050      209   3       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [26010]   500 26010     2879       95  15       0             0 tinyproxy
Apr 24 03:19:01 aidas-gsi kernel: [26513]     0 26513    25530      204  12       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [26517]   500 26517    25530      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29203]     0 29203    25016      164  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29207]   500 29207    25534      158   6       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 1105]   500  1105    27123      143   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17163]     0 17163    25016      190  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [17167]   500 17167    25534      185   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18690]     0 18690    25530      183   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18695]   500 18695    25530      138   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18696]   500 18696    27123      127   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [20802]     0 20802    25530      173   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20806]   500 20806    25530      142   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20807]   500 20807    27123      172   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 9697]     0  9697    25530      187   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9701]   500  9701    25563      184  10       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9703]   500  9703    27123      132   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [21737]   500 21737    29350      360  10       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [21860]   500 21860   166837     1623   5       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21861]   500 21861   208277     3415   0       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21868]   500 21868   470786     1884   2       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [20297]   500 20297   731145    44730   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [23452]     0 23452    47361      305   5       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [23453]     0 23453    25240       86   2       0             0 tail
Apr 24 03:19:01 aidas-gsi kernel: [23841]   500 23841    27123      243   3       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [  345]   500   345    27156      311  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 4825]   500  4825   682055    25908   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [16950]   500 16950    26921      257   1       0             0 watch
Apr 24 03:19:01 aidas-gsi kernel: [10822]   500 10822  1060110   432573   3       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [29109]   500 29109    27186      482   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 3754]   500  3754    27123      375   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11083]   500 11083    28030      216   3       0             0 HTTPD@8115
Apr 24 03:19:01 aidas-gsi kernel: [11087]   500 11087    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11089]   500 11089    36731     3211  12       0             0 tclsh8.5_copy2
Apr 24 03:19:01 aidas-gsi kernel: [11097]   500 11097    28030      207   1       0             0 TapeServer
Apr 24 03:19:01 aidas-gsi kernel: [11104]   500 11104     5398     1567  10       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [11105]   500 11105    25231      134  12       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11106]   500 11106     4323      187   6       0             0 stats
Apr 24 03:19:01 aidas-gsi kernel: [11107]   500 11107     4918      219   5       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11108]   500 11108     4917      194   9       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11109]   500 11109     4917      194  12       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11110]   500 11110     4630      214   0       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11111]   500 11111     4630      182   9       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11114]   500 11114    28030      217  12       0             0 NewMerger
Apr 24 03:19:01 aidas-gsi kernel: [11124]   500 11124   404629     8857   0       0             0 master64
Apr 24 03:19:01 aidas-gsi kernel: [11125]   500 11125    25231      135   9       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11126]   500 11126    19761    17318  13       0             0 statrate64
Apr 24 03:19:01 aidas-gsi kernel: [11127]   500 11127    35968    11059   2       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11128]   500 11128    35968    11498  11       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11129]   500 11129    35968    11581  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11130]   500 11130    35968    12366   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11131]   500 11131    35968    10031   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11132]   500 11132    35968    10689   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11133]   500 11133    35968    11896   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11134]   500 11134    35968    10510  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11135]   500 11135    35968    15679  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11136]   500 11136    35968    14623  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11137]   500 11137    35968    12790  15       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11138]   500 11138    35968    11683   0       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11139]   500 11139    35968     8773   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11140]   500 11140    35968     8774  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11141]   500 11141    35968     8772  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11142]   500 11142    35968     8773  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11152]   500 11152   307392    47667   5       0             0 merge64
Apr 24 03:19:01 aidas-gsi kernel: [11200]   500 11200   723346    37902   5       0             0 file:// Content
Apr 24 03:19:01 aidas-gsi kernel: [11338]   500 11338    28030      216   0       0             0 HTTPD
Apr 24 03:19:01 aidas-gsi kernel: [11342]   500 11342    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11344]   500 11344    46965    14788   3       0             0 tclsh8.5
Apr 24 03:19:01 aidas-gsi kernel: [12835]   500 12835   766232    71066   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [24071]   500 24071    27123      409   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [24347]   500 24347    28030      231   4       0             0 datarelaymbs
Apr 24 03:19:01 aidas-gsi kernel: [24351]   500 24351     7751     1313  10       0             0 DataRelayMBS
Apr 24 03:19:01 aidas-gsi kernel: [  407]    89   407    20280      470  13       0             0 pickup
Apr 24 03:19:01 aidas-gsi kernel: [ 1586]     0  1586     4238      156  13       0             0 anacron
Apr 24 03:19:01 aidas-gsi kernel: [ 1876]     0  1876    25231      144  10       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: Out of memory: Kill process 18393 (Xorg) score 523 or sacrifice child
Apr 24 03:19:01 aidas-gsi kernel: Killed process 18393, UID 0, (Xorg) total-vm:13906540kB, anon-rss:9558800kB, file-rss:92712kB
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11471" (uid=500 pid=1884 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11473" (uid=500 pid=1887 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:45 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:19:47 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1961 of process 1961 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Apr 24 03:19:48 aidas-gsi pulseaudio[1961]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1965 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1966 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1967 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11485" (uid=500 pid=1981 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11487" (uid=500 pid=1983 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: DNS-SD initialization failed: Daemon not running
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: init gpgme version 1.1.8
Apr 24 03:30:44 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2243 of process 2243 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:45 aidas-gsi pulseaudio[2243]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2301 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2302 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2303 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2325 of process 2325 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2325]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2326 of process 2326 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2326]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11506" (uid=500 pid=2328 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2418 of process 2418 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2418]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11508" (uid=500 pid=2420 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:51 aidas-gsi pulseaudio[2243]: ratelimit.c: 25 events suppressed
Apr 24 03:30:51 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr 24 03:33:11 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11521" (uid=500 pid=3005 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11524" (uid=500 pid=3012 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11525" (uid=500 pid=3015 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11527" (uid=500 pid=3018 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11529" (uid=500 pid=3022 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11531" (uid=500 pid=3024 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:13 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11533" (uid=500 pid=3100 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:37 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:40:46 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:41:22 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:19 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:43:31 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:47:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:816 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:952 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:884 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:895 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:892 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:839 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:897 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.8
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.1
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.7
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.3
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.4
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.2
Apr 24 03:51:04 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.5
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:29 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:06 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:814 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:852 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:694 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:770 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.9
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.10
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.12
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.11
Apr 24 03:52:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
  270   Sat Apr 24 03:13:05 2021 MLAIDA DAQ Reset

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

 

  269   Sat Apr 24 01:44:09 2021 MLsystem wide checks

system wide checks:

 

 

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

 

Attachment 1: 24-04-2021_230am_AIDA_HV.png
24-04-2021_230am_AIDA_HV.png
Attachment 2: 24-04-2021_230am_ucesbStat.png
24-04-2021_230am_ucesbStat.png
Attachment 3: 24-04-2021_230am_grafana.png
24-04-2021_230am_grafana.png
Attachment 4: 24-04-2021_230am_AidaGoodEvent.png
24-04-2021_230am_AidaGoodEvent.png
Attachment 5: 24-04-2021_230am_Correlation#8.png
24-04-2021_230am_Correlation#8.png
Attachment 6: 24-04-2021_230am_RESUME#3.png
24-04-2021_230am_RESUME#3.png
Attachment 7: 24-04-2021_230am_PAUSE#2.png
24-04-2021_230am_PAUSE#2.png
  268   Fri Apr 23 20:19:52 2021 TDFriday 23 April 20.00-00.00
21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378
Attachment 1: 201.png
201.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Attachment 4: Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Attachment 5: 202.png
202.png
Attachment 6: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
  267   Fri Apr 23 15:13:28 2021 DSJ23 April 16.00-20.00 shift

16:00 taking over from Muneerah

16:15 - temp, ucesb, database, stats etc. look ok.

16:41 - Nic restarting MBS

17:05 - temp, ucesb, database, stats etc. look ok.

17:43 - temp, ucesb, database, stats etc. look ok.

 

 

18.10 - system wide checks.  All same as previous checks - eg elog 264 + 266

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
 

Check FPGA Timestamp Error - HTML errors

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      7      4      7      4      3      2      4      2      3      6   : 36388
aida02 :      1      2      5      2      0      4      3      4      3      4      3   : 27044
aida03 :     28      4      4      4      4      4      2      2      3      3      6   : 36432
aida04 :      5      5      2      6      2      3      3      3      3      3      6   : 36892
aida05 :     24      4      7      3      3      3      2      2      3      3      6   : 36240
aida06 :     21      6      6      3      3      4      2      2      3      3      6   : 36356
aida07 :     15      7      4      3      3      4      2      2      3      3      6   : 36308
aida08 :     23      6      7      1      3      4      1      3      3      3      6   : 36572
aida09 :      1      8      2      1      1      2      4      2      3      3      6   : 36292
aida10 :      3      4      6      2      2      2      3      3      2      3      6   : 35660
aida11 :     15      7      4      1      2      3      2      4      2      3      6   : 36052
aida12 :     25     13      5      2      2      3      2      3      3      3      6   : 36700
 

18:45 - temp, ucesb, database, stats etc. look ok.

19:03 - temp, ucesb, database, stats etc. look ok.

19:26  - temp, ucesb, database, stats etc. look ok.

TapeData (df -h) - 7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7

 

19:30 - beam being shared with R3B, implantation rates fluctuation by~  50% back to normal 19:40

 

19:50 system checks - all as previous

 

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

    
         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Check FPGA Timestamp Error - HTML errors

 

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     25      6      4      7      4      3      2      4      2      3      6   : 36404
aida02 :     20      7      5      1      0      4      2      3      2      3      4   : 27384
aida03 :     28      6      3      5      3      5      2      2      3      3      6   : 36528
aida04 :     19      8      4      6      2      4      4      2      3      3      6   : 36876
aida05 :      7      3      8      3      4      3      1      3      3      3      6   : 36500
aida06 :     25      8      4      3      3      4      2      3      3      3      6   : 36868
aida07 :     18      7      3      4      3      4      2      2      3      3      6   : 36336
aida08 :     16     10      6      2      3      3      1      3      3      3      6   : 36464
aida09 :      0      3      1      1      1      3      4      2      3      3      6   : 36360
aida10 :      8      1     10      0      2      2      2      4      2      3      6   : 35912
aida11 :      2      2      2      3      3      4      3      3      2      3      6   : 35928
aida12 :     12      7      4      1      3      3      2      3      3      3      6   : 36616
 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi(5).png
Screenshot_2021-04-23_Statistics_aidas-gsi(5).png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(5).png
Attachment 3: 555.png
555.png
Attachment 4: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(6).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(6).png
Attachment 5: Screenshot_2021-04-23_Statistics_aidas-gsi(6).png
Screenshot_2021-04-23_Statistics_aidas-gsi(6).png
Attachment 6: 556.png
556.png
  266   Fri Apr 23 13:13:46 2021 MA, TDFriday 23 April 12.00-16.00
14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048
Attachment 1: 200.png
200.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Attachment 4: 210423_1550_Voltages.png
210423_1550_Voltages.png
Attachment 5: 210423_1550_Stats.png
210423_1550_Stats.png
Attachment 6: 210423_1550_Temp.png
210423_1550_Temp.png
  265   Fri Apr 23 12:12:08 2021 TDaida02 restart - 03.13UTC+2 Friday 23 April
Overnight aida02 crashed and restarted itself. A power cycle of all FEE64s was required to recover.

The system log appears to show the aida02 embedded processor crashed (note log times are UTC+1
for some reason).

Apr 21 00:41:21 aida02 kernel: xaida: open:
Apr 23 02:09:41 aida02 kernel: Unable to handle kernel paging request for data at address 0x00200200
Apr 23 02:09:41 aida02 kernel: Faulting instruction address: 0xc0091dd8
Apr 23 02:09:41 aida02 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 23 02:09:41 aida02 kernel: PREEMPT Xilinx Virtex440
Apr 23 02:09:41 aida02 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 23 02:09:42 aida02 kernel: NIP: c0091dd8 LR: c0091e40 CTR: 00000000
Apr 23 02:09:42 aida02 kernel: REGS: c6835e60 TRAP: 0300   Not tainted  (2.6.31)
Apr 23 02:09:42 aida02 kernel: MSR: 00021000 <ME,CE>  CR: 24000088  XER: 00000000
Apr 23 02:09:42 aida02 kernel: DEAR: 00200200, ESR: 00800000
Apr 23 02:09:42 aida02 kernel: TASK = c6824880[5] 'events/0' THREAD: c6834000
Apr 23 02:09:42 aida02 kernel: GPR00: c0091e40 c6835f10 c6824880 00000001 00000003 c038c3a8 00000000 c0498000
Apr 23 02:09:42 aida02 kernel: GPR08: 00000008 00100100 00000051 00200200 00485e6c 00005aa8 4805de90 00000001
Apr 23 02:09:42 aida02 kernel: GPR16: 00000000 00000000 4801d000 00000003 c031aef4 c02fb040 c02fb008 00000002
Apr 23 02:09:42 aida02 kernel: GPR24: c6834000 00100100 00200200 c680ae00 00000001 c680daf0 c680dae0 c6940020
Apr 23 02:09:42 aida02 kernel: NIP [c0091dd8] drain_freelist+0x54/0x110
Apr 23 02:09:42 aida02 kernel: LR [c0091e40] drain_freelist+0xbc/0x110
Apr 23 02:09:42 aida02 kernel: Call Trace:
Apr 23 02:09:42 aida02 kernel: [c6835f10] [c0091e40] drain_freelist+0xbc/0x110 (unreliable)
Apr 23 02:09:42 aida02 kernel: [c6835f40] [c0093158] cache_reap+0x104/0x138
Apr 23 02:09:42 aida02 kernel: [c6835f60] [c0048908] worker_thread+0x140/0x204
Apr 23 02:09:42 aida02 kernel: [c6835fc0] [c004cfc8] kthread+0x78/0x7c
Apr 23 02:09:42 aida02 kernel: [c6835ff0] [c000e140] kernel_thread+0x4c/0x68
Apr 23 02:09:42 aida02 kernel: Instruction dump:
Apr 23 02:09:43 aida02 kernel: 3ba40010 7f80e800 419e00d4 3f200010 3f400020 54380024 63390100 635a0200
Apr 23 02:09:43 aida02 kernel: 3b800000 48000068 813f0000 817f0004 <912b0000> 91690004 933f0000 935f0004
Apr 23 02:09:43 aida02 kernel: ---[ end trace 6baa5b82e4ef4e92 ]---
Apr 23 02:09:43 aida02 kernel: note: events/0[5] exited with preempt_count 1
Apr 23 02:13:39 aida02 syslogd 1.4.2: restart.
Apr 23 02:13:39 aida02 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Apr 23 02:13:39 aida02 kernel: Using Xilinx Virtex440 machine description
Apr 23 02:13:39 aida02 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
  264   Fri Apr 23 07:09:57 2021 LJWFriday 23rd April 08:00-12:00

08:30

System Checks

 

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:


         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

 

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

 

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

 

10:45

System Checks

Clock Status Error:


FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


ADC Calibration check Error:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

 

Attachment 1: 2304211_1.png
2304211_1.png
Attachment 2: 230421_2.png
230421_2.png
Attachment 3: 230421_3.png
230421_3.png
Attachment 4: 230421_4.png
230421_4.png
Attachment 5: 230421_5.png
230421_5.png
Attachment 6: 230421_6.png
230421_6.png
Attachment 7: 230421_7.png
230421_7.png
Attachment 8: 230421_8.png
230421_8.png
Attachment 9: 230421_9.png
230421_9.png
  263   Thu Apr 22 23:17:46 2021 DJ Friday 23rd April 00:00-08:00

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

 

 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi.png
Screenshot_2021-04-23_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot-CAEN_HV_Supply.png
Screenshot-CAEN_HV_Supply.png
Attachment 4: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Attachment 5: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 60.png
60.png
Attachment 10: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Attachment 12: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Attachment 13: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Attachment 17: 61.png
61.png
Attachment 18: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Attachment 19: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Attachment 20: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Attachment 21: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 22: Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Attachment 23: 63.png
63.png
Attachment 24: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Attachment 25: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Attachment 26: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Attachment 27: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
  262   Thu Apr 22 15:18:20 2021 TDThursday 22 April 16.00-00.00
16.20 zero all histograms, statistics & merger statistics

      most recent merger error messages

MERGE Data Link (29984): bad timestamp  6 3 0xc1ac7f1c 0x0dce7672 0x00002cda5dce7672 0x16782cda5dce7672 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc18b7e62 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ebc 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1b67ecb 0x0dcf0ae2 0x00002cda5dcf0ae2 0x16782cda5dcf0ae2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ea3 0x0dcf12b2 0x00002cda5dcf12b2 0x16782cda5dcf12b2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1867fba 0x0b2ee142 0x00002dc9eb2ee142 0x16782dc9eb2ee142 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0x86800889 0x0ad28158 0x00002dc9eb2ee142 0x16782dc9ead28158 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0xc1be7e6e 0x0b2ef0e2 0x00002dc9eb2ef0e2 0x16782dc9eb2ef0e2 0x16782dc9fad2c172
MERGE Data Link (29984): bad timestamp  6 3 0xc1b08006 0x0b2f5e42 0x00002dc9eb2f5e42 0x16782dc9eb2f5e42 0x16782dc9fad34502
MERGE Data Link (29984): bad timestamp  6 3 0xc1807de7 0x0ad3c342 0x00002dc9eb2f6de2 0x16782dc9ead3c342 0x16782dc9fad34502
MERGE Statistics counters now cleared

16.21 merger server terminal reset & clear

      TapeData -> /media/ThirdDRive/TapeData

[npg@aidas-gsi bin64]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   13G   34G  28% /
tmpfs                 7.9G  559M  7.3G   7% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   79G  302G  21% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  6.3T  605G  92% /media/SecondDrive
/dev/sdc1             7.2T  3.6T  3.3T  52% /media/ThirdDrive

      at current rate (10.5Mb/s) we have 3.6d remaining 
      
17.40 *all* system wide checks OK *except* FPGA timestamp errors

		 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x51 : 	 81  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

17.43 FEE64 temps OK - attachment 1
      statistics - attachments 2-8
      per FEE64 rate spectra - attachments 9 & 10
      1.8.L spectra - attachments 11-14
      1.8.H spectra - attachments 15-16
      1.8.W spectra 20us FSR - attachments 17-18
      1.8.W spectra 200us, 2ms & 20ms FSR - attachments 19-21, 30-32
      merger, tape server & merger statistics 19-25
      ucesb - attachment 26
      Grafana DSSD biases & leakage currents - attachment 27
      taperserver & tapeserver/merger server terminals - no timewarp errors since c. 15.25 - attachment 28
      DSSSD biases & leakage currents - attachment 29


22.17 *all* system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x3612 : 	 0x3613 : 	 1  
aida02 fault 	 0x523c : 	 0x523d : 	 1  
aida03 fault 	 0xc5cd : 	 0xc5ce : 	 1  
aida04 fault 	 0x14f1 : 	 0x14f2 : 	 1  
aida05 fault 	 0x9a4e : 	 0x9a4f : 	 1  
aida06 fault 	 0xfdd2 : 	 0xfdd3 : 	 1  
aida07 fault 	 0x4a0 : 	 0x4a1 : 	 1  
aida08 fault 	 0xc796 : 	 0xc797 : 	 1  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x57 : 	 87  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors

MERGE Data Link (29979): bad timestamp  1 3 0xc05080ef 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc06f80d7 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0778222 0x0c88b272 0x000047aa2c88b272 0x167847aa2c88b272 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0598098 0x0c88e152 0x000047aa2c88e152 0x167847aa2c88e152 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc042815b 0x0c894eb2 0x000047aa2c894eb2 0x167847aa2c894eb2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc05980eb 0x0c899502 0x000047aa2c899502 0x167847aa2c899502 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc075813c 0x0c89e322 0x000047aa2c89e322 0x167847aa2c89e322 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc04280d5 0x0c89f2c2 0x000047aa2c89f2c2 0x167847aa2c89f2c2 0x167847aa2c8a34d2
MERGE Actor (29995): Working with 0 from 12 data sources
MERGE Actor (29995): Working with 0 from 12 data sources

22.20 DSSSD biases & leakage currents - attachment 33
      FEE64 temps OK - attachment 34
      statistics - attachments 35
      merger statistics 36

      
Attachment 1: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 2: Screenshot_2021-04-22_Statistics_aidas-gsi(4).png
Screenshot_2021-04-22_Statistics_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi(5).png
Screenshot_2021-04-22_Statistics_aidas-gsi(5).png
Attachment 4: Screenshot_2021-04-22_Statistics_aidas-gsi(6).png
Screenshot_2021-04-22_Statistics_aidas-gsi(6).png
Attachment 5: Screenshot_2021-04-22_Statistics_aidas-gsi(7).png
Screenshot_2021-04-22_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-04-22_Statistics_aidas-gsi(8).png
Screenshot_2021-04-22_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-22_Statistics_aidas-gsi(9).png
Screenshot_2021-04-22_Statistics_aidas-gsi(9).png
Attachment 8: Screenshot_2021-04-22_Statistics_aidas-gsi(10).png
Screenshot_2021-04-22_Statistics_aidas-gsi(10).png
Attachment 9: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Attachment 10: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 11: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 12: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(4).png
Attachment 14: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(5).png
Attachment 15: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(6).png
Attachment 16: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(7).png
Attachment 17: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(8).png
Attachment 18: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(9).png
Attachment 19: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(10).png
Attachment 20: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(11).png
Attachment 21: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(12).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(12).png
Attachment 22: Screenshot_2021-04-22_NewMerger_Control_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Control_aidas-gsi.png
Attachment 23: Screenshot_2021-04-22_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-22_Tape_Service_(Expert)_aidas-gsi.png
Attachment 24: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Attachment 25: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi(1).png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi(1).png
Attachment 26: Screenshot_2021-04-22_ucesb.png
Screenshot_2021-04-22_ucesb.png
Attachment 27: Screenshot_2021-04-22_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-04-22_AIDA_Alerting_-_Grafana(1).png
Attachment 28: 101.png
101.png
Attachment 29: 100.png
100.png
Attachment 30: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Attachment 31: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 32: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 33: 102.png
102.png
Attachment 34: Screenshot_2021-04-22_Statistics_aidas-gsi.png
Screenshot_2021-04-22_Statistics_aidas-gsi.png
Attachment 35: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 36: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
  261   Thu Apr 22 11:03:57 2021 JSThursday 22nd April: 12:00-16:00
12:00 Taking over from Liam, all is well.

12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz

12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they 
need to check and will need apprx. 15 min.")

12:30
ucesb ok (no beam)
DB No faults
Stats ok.

12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.

13:00
We are taking intermittent beam while they work on the FRS

DB No faults
ucesb ok  elog:261/1
Current ok elog:261/2
Stats ok  elog:261/3
Temps ok elog:261/4

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x46 : 	 70  
FPGA Timestamp error counter test result: Passed 11, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      6      5      3      2      2      2      3      2      3      7   : 39580
aida02 :     21      7      4      2      2      3      2      3      2      3      7   : 39692
aida03 :     20     10      3      3      0      3      2      3      2      3      7   : 39600
aida04 :      8      7      4      5      2      4      3      2      2      3      7   : 39608
aida05 :     17     14      1      2      3      3      4      2      2      3      7   : 39748
aida06 :     17     10      2      3      2      2      4      2      2      3      7   : 39572
aida07 :     27     10      8      7      2      3      3      2      2      3      7   : 39708
aida08 :      9      6      2      3      2      1      3      3      2      3      7   : 39636
aida09 :      1      2      1      4      0      3      3      2      2      3      7   : 39204
aida10 :      5      8      4      4      2      2      1      3      2      3      7   : 39316
aida11 :      6      4      5      3      2      3      2      2      2      3      7   : 39144
aida12 :     15      5      8      1      0      4      1      3      2      3      7   : 39428


13:32
ucesb ok
DB No faults
Stats ok

14:01
ucesb ok
DB No faults
Stats ok

14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
      cat /var/log/messages | grep /MIDAS - attachments 6 & 7
      Grafana DSSSD bias & leakage currents for last 7 days - attachment 8

14:55
DB No faults 
ucesb ok
Current ok elog:261/9
Stats ok  elog:261/10
Temps ok elog:261/11

FPGA Timestamp Check Error:
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 11, Failed 1


15:30
ucesb ok
DB No faults
Stats ok

15:59
ucesb ok elog:261/12
DB No faults
Stats ok
Attachment 1: 51.png
51.png
Attachment 2: 52.png
52.png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi(2).png
Screenshot_2021-04-22_Statistics_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 5: R50_528
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256073749 ( 1415213.2 Hz)
        Other data format:    5846251 (   32309.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32674 (     180.6 Hz)
                              WR48-63:      32674 (     180.6 Hz)
                           FEE64 disc:    1081787 (    5978.6 Hz)
                             MBS info:    4699116 (   25970.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     342915 (    1895.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      180.944 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               53.074
  2                0.000               66.167
  3                0.000               47.044
  4                0.000              162.945
  5                0.000              122.260
  6                0.000              104.653
  7                0.000              108.241
  8                0.000               75.998
  9                0.000                7.078
 10                0.000               27.522
 11                0.000               39.393
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18754736     101009          0          0          0          0       2455       2455      95016       1083          0      35321
  1   22221860      61355          0          0          0          0       2741       2741      55873          0          0      20378
  2   16643754    1970924          0          0          0          0       2367       2367     197444    1768746          0      33880
  3   21495935    1857330          0          0          0          0       2886       2886      82623    1768935          0      33490
  4    3525133     403992          0          0          0          0        456        456      99315     303765          0      35967
  5   10189380     355501          0          0          0          0       1262       1262      56604     296373          0      20229
  6   12085720     387518          0          0          0          0       1592       1592     108226     276108          0      33721
  7   11376753     370078          0          0          0          0       1497       1497      82978     284106          0      32056
  8   43639711      66166          0          0          0          0       5406       5406      55354          0          0      18854
  9   42033424      67528          0          0          0          0       5337       5337      56854          0          0      18886
 10   29147293     115191          0          0          0          0       3615       3615     107961          0          0      31625
 11   24960050      89659          0          0          0          0       3060       3060      83539          0          0      28508
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:52321.719s (    0.612 blocks/s,   0.038 Mb/s)
Attachment 6: log2
Apr 13 17:43:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:40 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:45:01 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:43 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:08 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:12 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:33:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Attachment 7: log1
Apr 19 08:33:03 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:37 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:40 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:48 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:48 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:51 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:51 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:53 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:53 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:30:00 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:33:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:26 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:27 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:43 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:45 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:45 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:04:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:55 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:57 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:10:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:01 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:05 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:06 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:13 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:19 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:19 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:26 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:21:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:21:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:28 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:50:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:08:07 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:55 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:33:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:07:57 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:21 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:22 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:28 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:47 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:40:07 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Attachment 8: Screenshot_2021-04-22_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-22_AIDA_Alerting_-_Grafana.png
Attachment 9: 53.png
53.png
Attachment 10: Screenshot_2021-04-22_Statistics_aidas-gsi(3).png
Screenshot_2021-04-22_Statistics_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 12: 54.png
54.png
  260   Thu Apr 22 07:06:37 2021 LPGThursday 22nd April: 08:00-12:00
08:01 - Taking over from Corrigan...

ucesb ok!
DB ok!
Stats ok!

08:30

ucesb ok!
DB ok!
Stats ok!

09:15

ucesb ok! elog:260/4
DB ok!
Stats ok!

Leakage currents ok! elog:260/1
Temperatures ok! elog:260/2
MIDAS Rates ok! elog:260/3

System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x40 : 64
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!


09:45

ucesb ok!
DB ok!
Stats ok!

10:30

ucesb ok!
DB ok!
Stats ok!


11:00

ucesb ok! elog:260/5
DB ok!
Stats ok!

Leakage currents ok! elog:260/6
Temperatures ok! elog:260/7
MIDAS Rates ok! elog:260/8

System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x42 : 66
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!


11:30

ucesb ok! elog:260/5
DB ok!
Stats ok!

11:55

ucesb ok!
DB ok!
Stats ok!
Attachment 1: lpg20.png
lpg20.png
Attachment 2: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi.png
Screenshot_2021-04-22_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-22_at_08.19.51.png
Screenshot_2021-04-22_at_08.19.51.png
Attachment 5: Screenshot_2021-04-22_at_09.57.10.png
Screenshot_2021-04-22_at_09.57.10.png
Attachment 6: lpg21.png
lpg21.png
Attachment 7: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-22_Statistics_aidas-gsi(1).png
Screenshot_2021-04-22_Statistics_aidas-gsi(1).png
  259   Wed Apr 21 23:05:39 2021 CA22nd April 00:00 - 08:00
00:07 stats/ucesb/DB checks all ok

00:30 all system wide checks ok *except* fpga errors

      			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1f : 	 31  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     statistics ok - attachment 1

     temperatures ok - attachment 2

     detector bias / leakage currents ok - attachment 3

     no error messages in DB terminal & ucesb ok - attachment 4

01:00 stats/ucesb/DB checks all ok

01:30 stats/ucesb/DB checks all ok

01:53 issues with beam dropping in and out - ESR working to stabilise this

02:00 stats/ucesb/DB checks all ok

02:09 beam back in a stable state

02:30 all system wide checks ok *except* fpga errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:33 statistics ok - attachment 5

      temperatures ok - attachment 6

      detector bias / leakage currents ok - attachment 7

      no error messages in DB terminal & ucesb ok - attachment 8

02:45 beam off - potentially serious issue, awaiting update from FRS

03:00 significant issues with UNILAC, beam not expected back for a while

03:05 stats/ucesb/DB checks all ok

03:32 stats/ucesb/DB checks all ok

03:55 beam is back

04:00 stats/ucesb/DB checks all ok

04:30 all system wide checks ok *except* fpga errors

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x2b : 	 43  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:32 statistics ok - attachment 9

      temperatures ok - attachment 10

      detector bias / leakage currents ok - attachment 11

      no error messages in DB terminal & ucesb ok - attachment 12

05:00 stats/ucesb/DB checks all ok

05:30 stats/ucesb/DB checks all ok

06:00 stats/ucesb/DB checks all ok

06:30 all system wide checks ok *except* fpga errors

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x34 : 	 52  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

06:32 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias / leakage currents ok - attachment 15

      no error messages in DB terminal & ucesb ok - attachment 16

07:00 stats/ucesb/DB checks all ok

07:30 stats/ucesb/DB checks all ok

07:59 stats/ucesb/DB checks all ok



      
Attachment 1: Screenshot_from_2021-04-21_23-31-22.png
Screenshot_from_2021-04-21_23-31-22.png
Attachment 2: Screenshot_from_2021-04-21_23-31-44.png
Screenshot_from_2021-04-21_23-31-44.png
Attachment 3: Screenshot_from_2021-04-21_23-32-03.png
Screenshot_from_2021-04-21_23-32-03.png
Attachment 4: Screenshot_from_2021-04-21_23-34-46.png
Screenshot_from_2021-04-21_23-34-46.png
Attachment 5: Screenshot_from_2021-04-22_01-31-14.png
Screenshot_from_2021-04-22_01-31-14.png
Attachment 6: Screenshot_from_2021-04-22_01-31-37.png
Screenshot_from_2021-04-22_01-31-37.png
Attachment 7: Screenshot_from_2021-04-22_01-34-03.png
Screenshot_from_2021-04-22_01-34-03.png
Attachment 8: Screenshot_from_2021-04-22_01-34-31.png
Screenshot_from_2021-04-22_01-34-31.png
Attachment 9: Screenshot_from_2021-04-22_03-32-36.png
Screenshot_from_2021-04-22_03-32-36.png
Attachment 10: Screenshot_from_2021-04-22_03-32-57.png
Screenshot_from_2021-04-22_03-32-57.png
Attachment 11: Screenshot_from_2021-04-22_03-33-11.png
Screenshot_from_2021-04-22_03-33-11.png
Attachment 12: Screenshot_from_2021-04-22_03-33-57.png
Screenshot_from_2021-04-22_03-33-57.png
Attachment 13: Screenshot_from_2021-04-22_05-33-01.png
Screenshot_from_2021-04-22_05-33-01.png
Attachment 14: Screenshot_from_2021-04-22_05-33-19.png
Screenshot_from_2021-04-22_05-33-19.png
Attachment 15: Screenshot_from_2021-04-22_05-33-30.png
Screenshot_from_2021-04-22_05-33-30.png
Attachment 16: Screenshot_from_2021-04-22_05-33-43.png
Screenshot_from_2021-04-22_05-33-43.png
  258   Wed Apr 21 15:01:14 2021 LJW, MAWednesday 21st April 16:00-20:00

16:30

All System checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1b :      27  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Temperature Checks: See attachment 1

Statistics Checks: See attachment 2

Bias & Leakage Currents: See attachment 3

 

18:09

All system checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Temperature checks: See attachment 4

Statistics Checks: See attachment 5

Bias & Leakage Currents: See attachment 6

 

19:40

All system checks okay, except for:

FPGA Timestamp error:

             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Temperature checks: See attachment 7

Statistics Checks: See attachment 8

Bias & Leakage Currents: See attachment 9

 

*Beam being tuned to another experiment (R3B) - all DSSD implants fluctuating back and forth to 0 Hz for around the last hour or so*

22:30

General check

Rates, Voltages, Temperatures, Ucesb are attached 10,11,12,13

Its been noticed that aida04 temp. is incresing to about 65 (red), still below 70.

the system wide check are all ok except FPGA


             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: 210421_1.png
210421_1.png
Attachment 2: 210421_2.png
210421_2.png
Attachment 3: 2104211_3.png
2104211_3.png
Attachment 4: 210421_4.png
210421_4.png
Attachment 5: 210421_5.png
210421_5.png
Attachment 6: 210421_6.png
210421_6.png
Attachment 7: 210421_7.png
210421_7.png
Attachment 8: 210421_8.png
210421_8.png
Attachment 9: 210421_9.png
210421_9.png
Attachment 10: 210421_2230_Stats.png
210421_2230_Stats.png
Attachment 11: 210421_2230_Temp.png
210421_2230_Temp.png
Attachment 12: 210421_2230_Ucesb.png
210421_2230_Ucesb.png
Attachment 13: 210421_2230_Voltages.png
210421_2230_Voltages.png
  257   Wed Apr 21 11:27:55 2021 OHWed 21st April 12:00-16:00
12:30 MBS has crashed
      This time AIDA is still running fine.

      Stats ok - attachment 1
      Temperature - attachment 2
      Bias and leakage currents - attachment 3
     
      All system wide checks ok *except*
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x19 : 	 25  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Now also note errors on aida7 and aida03 in the merger TS errors
     (0 represents the sum across all FEE)
0	1355800	0x0014b018	
1	0	0x00000000	
2	0	0x00000000	
3	33	0x00000021
4	0	0x00000000	
5	0	0x00000000	
6	0	0x00000000	
7	10	0x0000000a
8	0	0x00000000	
9	0	0x00000000	
10	0	0x00000000	
11	1355757	0x0014afed

15:30 CA takes over

      Stats ok - attachment 4
      Temperature - attachment 5
      Bias and leakage currents - attachment 6

      all system wide checks ok *except*

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x19 : 	 27  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:42 db check and ucesb ok
Attachment 1: 210421_1338_Stats.png
210421_1338_Stats.png
Attachment 2: 210421_1328_Temp.png
210421_1328_Temp.png
Attachment 3: 210421_1342_Bias.png
210421_1342_Bias.png
Attachment 4: Screenshot_from_2021-04-21_14-33-36.png
Screenshot_from_2021-04-21_14-33-36.png
Attachment 5: Screenshot_from_2021-04-21_14-34-29.png
Screenshot_from_2021-04-21_14-34-29.png
Attachment 6: Screenshot_from_2021-04-21_14-36-08.png
Screenshot_from_2021-04-21_14-36-08.png
  256   Wed Apr 21 06:56:42 2021 MSWednesday 21 April 08:00-12:00 2021

8:00


 

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      6      7      2      2      4      3      2      2      3      7   : 39596
aida02 :     25      3      2      1      1      3      2      3      2      3      7   : 39548
aida03 :     28      5      3      1      2      2      2      3      2      3      7   : 39528
aida04 :     27      6      2      2      2      4      2      4      2      3      7   : 40316
aida05 :      9      9      4      3      1      3      2      3      3      3      7   : 40652
aida06 :     21      7      6      1      2      2      4      2      2      3      7   : 39564
aida07 :     20      7      5      0      1      2      2      3      2      3      7   : 39448
aida08 :     21      5      4      1      1      2      3      3      2      3      7   : 39708
aida09 :      1      3      1      3      0      2      1      4      2      3      7   : 39564
aida10 :     20      6      3      2      1      2      3      3      2      3      7   : 39728
aida11 :     18      4      2      1      2      2      2      2      2      3      7   : 38952
aida12 :     19      8      7      1      1      2      3      3      2      3      7   : 39772

 

From the merger statistics during the night there have been 1355757 TS errors in aida12. Corrigan said they have been quiet since around 3am though.



10:00

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 1, Failed 11

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     15      7      6      2      1      4      3      2      2      3      7   : 39508
aida02 :      1      2      1      0      1      2      2      4      2      3      7   : 39780
aida03 :     23      8      2      0      1      3      2      3      2      3      7   : 39548
aida04 :     23      9      1      2      3      3      2      4      2      3      7   : 40244
aida05 :     14      7      2      2      4      3      2      3      1      4      7   : 40784
aida06 :     18      5      5      2      2      2      3      3      2      3      7   : 39808
aida07 :     27      4      3      2      2      3      3      2      2      3      7   : 39420
aida08 :     23      6      3      2      2      2      3      3      2      3      7   : 39804
aida09 :     18      4      6      3      2      4      3      2      2      3      7   : 39592
aida10 :      6      9      5      3      2      3      2      3      2      3      7   : 39696
aida11 :     16      4      1      2      3      4      1      3      2      3      7   : 39536
aida12 :     18      5      7      1      0      3      3      3      2      3      7   : 39808

 

10:22

NH Noticed that the ASIC clocks had not been synchronised following the last AIDA restart. - attachment 7
As the beam is having issues they are not currently recording
I have now synchronised the ASIC clocks.
All ADC are now calibrated

 

11:06

Used the beam off time to change the drive writing to. Now writing to /media/ThirdDrive/TapeData/S460/R50

11:10

Beam is back they will start writing to file again


12:00

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x17 :      23  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      5      5      1      1      4      2      3      2      3      7   : 39720
aida02 :     16      8      4      1      0      2      1      4      2      3      7   : 39648
aida03 :      9      7      3      2      3      3      2      3      2      3      7   : 39692
aida04 :      8      7      5      2      3      5      3      2      2      3      7   : 39720
aida05 :     20      5      3      4      2      3      3      3      2      3      7   : 39976
aida06 :     25      8      0      3      1      2      3      3      2      3      7   : 39748
aida07 :     24      8      4      2      3      3      2      3      2      3      7   : 39776
aida08 :     22      9      0      2      2      2      3      3      2      3      7   : 39776
aida09 :      1      4      4      3      2      3      2      3      2      3      7   : 39620
aida10 :     13      4      4      3      2      3      1      3      2      3      7   : 39412
aida11 :      2      3      1      1      3      4      2      2      2      3      7   : 39184
aida12 :     16     10      7      2      1      3      1      3      2      3      7   : 39424

 

Attachment 1: Screenshot_2021-04-21_Statistics_aidas-gsi.png
Screenshot_2021-04-21_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 46.png
46.png
Attachment 4: Screenshot_2021-04-21_Statistics_aidas-gsi(1).png
Screenshot_2021-04-21_Statistics_aidas-gsi(1).png
Attachment 5: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 6: 47.png
47.png
Attachment 7: Capture.PNG
Capture.PNG
Attachment 8: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-21_Statistics_aidas-gsi(2).png
Screenshot_2021-04-21_Statistics_aidas-gsi(2).png
Attachment 10: 48.png
48.png
  255   Tue Apr 20 23:08:22 2021 CAWed Apr 21 00:00-08:00
00:00 - CA takes over

00:15 - stats/database/ucesb ok - attachment 1

      - another burst of bad timestamp error messages in NewMerger terminal - but all FEE64 still ok - attachment 2

      - Merger/TapeService ok

00:21 - beam still off while UNILAC having problems

00:26 - beam is back - rate spectra - attachment 3

00:32 - ucesb following beam return - attachment 4

      - implant rates reaching highs of ~1kHz

00:35 - Grafana leakage current monitor last 6 hours - attachment 5

01:00 *all* system wide checks ok

      stats ok - attachment 6

      temperatures ok - attachment 7

      detector bias / leakage currents ok - attachment 8

      db check ok / ucesb ok

      still observe periodic bursts of bad timestamp errors in NewMerger terminal - FEE64s still linked with merger ok

01:30 AIDA crashes just as I make check, most FEE64 lost connection -> powercycle and restart MIDAS

01:53 AIDA reset completed - writing to file NULL/R48

01:56 all system wide checks ok *except* 

FPGA error counter

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


02:01 good event statistics ok - attachment 9

      temperatures ok - attachment 10

      detector bias / leakage currents ok - attachment 11

      db check / ucesb ok

02:20 beam stop

02:21 beam back

02:30 stats/ucesb/db checks all ok

03:00 stats/ucesb/db checks all ok - attachment 12

03:30 stats/ucesb/db checks all ok

04:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:03 beam off briefly

04:05 good event statistics ok - attachment 13

04:07 beam back

      temperatures ok - attachment 14

      bias / leakage currents ok - attachment 15

      db check / ucesb ok - attachment 16

      implant rates in AIDA peaking at ~1.5kHz

04:30 stats/ucesb/db checks all ok

04:48 no beam

04:58 beam back

05:00 stats/ucesb/db checks all ok - attachment 17

05:30 stats/ucesb/db checks all ok

06:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

    FEE64 Linux Memory Info

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      5      3      3      2      3      2      3      2      3      7   : 39628
aida02 :     22     11      3      0      2      3      2      3      2      3      7   : 39648
aida03 :     20     10      1      1      2      2      1      4      2      3      7   : 39760
aida04 :     17     10      2      2      3      3      2      3      2      3      7   : 39732
aida05 :      1      8      3      2      3      2      2      3      3      3      7   : 40564
aida06 :      5      6      2      3      2      2      3      3      2      3      7   : 39748
aida07 :     18      4      7      0      1      3      1      4      2      3      7   : 39832
aida08 :     22     10      3      3      2      2      3      3      2      3      7   : 39864
aida09 :      3      6      2      2      0      3      1      4      2      3      7   : 39708
aida10 :      2      4      3      2      1      2      2      3      2      3      7   : 39384
aida11 :     20      8      3      1      1      3      3      2      2      3      7   : 39328
aida12 :     16      6      3      1      0      3      2      3      2      3      7   : 39488

06:02 good event statistics ok - attachment 18

      FEE64 temperatures ok - attachment 19

      detector bias / leakage currents ok - attachment 20

06:06 db check and ucesb ok - attachment 21

      NewMerger terminal hasn't flagged up any more errors since ~ 03:00

06:30 stats/ucesb/db checks all ok

      DESPEC restarts timesorter

07:00 stats/ucesb/db checks all ok

07:30 stats/ucesb/db checks all ok - attachment 22

      grafana leakage currents last 1 hour - attachment 23

   

    







      
Attachment 1: Screenshot_from_2021-04-20_23-15-20.png
Screenshot_from_2021-04-20_23-15-20.png
Attachment 2: Screenshot_from_2021-04-20_23-18-14.png
Screenshot_from_2021-04-20_23-18-14.png
Attachment 3: Screenshot_from_2021-04-20_23-29-06.png
Screenshot_from_2021-04-20_23-29-06.png
Attachment 4: Screenshot_from_2021-04-20_23-32-08.png
Screenshot_from_2021-04-20_23-32-08.png
Attachment 5: Screenshot_from_2021-04-20_23-34-42.png
Screenshot_from_2021-04-20_23-34-42.png
Attachment 6: Screenshot_from_2021-04-21_00-00-06.png
Screenshot_from_2021-04-21_00-00-06.png
Attachment 7: Screenshot_from_2021-04-21_00-00-29.png
Screenshot_from_2021-04-21_00-00-29.png
Attachment 8: Screenshot_from_2021-04-21_00-01-15.png
Screenshot_from_2021-04-21_00-01-15.png
Attachment 9: Screenshot_from_2021-04-21_01-00-03.png
Screenshot_from_2021-04-21_01-00-03.png
Attachment 10: Screenshot_from_2021-04-21_01-00-48.png
Screenshot_from_2021-04-21_01-00-48.png
Attachment 11: Screenshot_from_2021-04-21_01-01-05.png
Screenshot_from_2021-04-21_01-01-05.png
Attachment 12: Screenshot_from_2021-04-21_01-59-30.png
Screenshot_from_2021-04-21_01-59-30.png
Attachment 13: Screenshot_from_2021-04-21_03-05-24.png
Screenshot_from_2021-04-21_03-05-24.png
Attachment 14: Screenshot_from_2021-04-21_03-05-47.png
Screenshot_from_2021-04-21_03-05-47.png
Attachment 15: Screenshot_from_2021-04-21_03-06-07.png
Screenshot_from_2021-04-21_03-06-07.png
Attachment 16: Screenshot_from_2021-04-21_03-11-44.png
Screenshot_from_2021-04-21_03-11-44.png
Attachment 17: Screenshot_from_2021-04-21_04-00-22.png
Screenshot_from_2021-04-21_04-00-22.png
Attachment 18: Screenshot_from_2021-04-21_05-00-18.png
Screenshot_from_2021-04-21_05-00-18.png
Attachment 19: Screenshot_from_2021-04-21_05-00-43.png
Screenshot_from_2021-04-21_05-00-43.png
Attachment 20: Screenshot_from_2021-04-21_05-03-47.png
Screenshot_from_2021-04-21_05-03-47.png
Attachment 21: Screenshot_from_2021-04-21_05-05-43.png
Screenshot_from_2021-04-21_05-05-43.png
Attachment 22: Screenshot_from_2021-04-21_06-30-20.png
Screenshot_from_2021-04-21_06-30-20.png
Attachment 23: Screenshot_from_2021-04-21_06-32-51.png
Screenshot_from_2021-04-21_06-32-51.png
  254   Tue Apr 20 19:00:00 2021 JSTuesday 20th 20:00-00:00
20:00  

Taking over from Liam, all systems good.

20:30  

DB No faults.
Stats good
ucesb ok

20:58 

DB No faults.
Stats good
Temps good
ucesb ok

21:31 

DB No faults.
Stats good
Temps good
ucesb ok

22:02

DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2

ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1

22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5

Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193

22:50
After a full power cycle all appear to be up and running well. Starting full checks.

23:00

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

Check FPGA is having a page error

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	
4k
aida01 :      4      8      2      2      1      3      1      4      2      3      7   : 
39792
aida02 :     13      9      2      4      2      3      2      3      2      3      7   : 
39708
aida03 :     19      5      4      3      1      3      1      4      2      3      7   : 
39892
aida04 :     14      5      2      2      0      5      2      3      2      3      7   : 
39744
aida05 :     16     11      3      2      2      4      3      2      1      4      7   : 
40584
aida06 :     15      9      5      2      2      2      1      4      2      3      7   : 
39828
aida07 :     15      9      3      3      2      2      1      4      2      3      7   : 
39828
aida08 :     21      4      3      4      2      2      1      4      2      3      7   : 
39844
aida09 :     11      6      1      2      3      4      2      3      2      3      7   : 
39788
aida10 :      0      6      4      2      0      3      2      3      2      3      7   : 
39472
aida11 :     16      7      2      0      1      3      2      3      2      3      7   : 
39512
aida12 :      3      7      4      1      3      2      2      3      2      3      7   : 
39524

DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok

23:25 

DB No faults.
Stats good
Temps good
ucesb ok

23:42 No Beam
23:48 Beam Back

23:57

DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
	
*FPGA check page error*

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      9      5      2      1      2      1      4      2      3      7   : 39812
aida02 :     13      9      5      2      2      2      1      4      2      3      7   : 39820
aida03 :     21      8      3      2      2      2      1      4      2      3      7   : 39812
aida04 :     13      4      3      1      1      5      2      3      2      3      7   : 39780
aida05 :     24      5      3      2      2      4      2      3      2      3      7   : 39800
aida06 :     16      5      5      2      2      2      1      4      2      3      7   : 39800
aida07 :     22      6      2      2      1      3      1      4      2      3      7   : 39848
aida08 :     21      8      2      3      2      2      1      4      2      3      7   : 39828
aida09 :     18      7      1      1      3      4      2      3      2      3      7   : 39792
aida10 :     19      3      3      1      0      3      2      3      2      3      7   : 39476
aida11 :     17      4      4      0      1      3      2      3      2      3      7   : 39524
aida12 :     13      4      3      2      3      1      2      3      2      3      7   : 39428

00:04 Handing over to Corrigan
Attachment 1: 40.png
40.png
Attachment 2: 41.png
41.png
Attachment 3: Screenshot_2021-04-20_Statistics_aidas-gsi(3).png
Screenshot_2021-04-20_Statistics_aidas-gsi(3).png
Attachment 4: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-20_Statistics_aidas-gsi(4).png
Screenshot_2021-04-20_Statistics_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-20_Statistics_aidas-gsi(5).png
Screenshot_2021-04-20_Statistics_aidas-gsi(5).png
Attachment 7: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 8: 42.png
42.png
Attachment 9: 44.png
44.png
Attachment 10: 45.png
45.png
Attachment 11: Screenshot_2021-04-20_Statistics_aidas-gsi(6).png
Screenshot_2021-04-20_Statistics_aidas-gsi(6).png
Attachment 12: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(5).png
  253   Tue Apr 20 14:56:38 2021 LPGTuesday 20th 16:00-20:00
16:00

New shift, ready to go! Beam stopped just as I started... Waiting for update.
Beam back at 16:08, ucesb scalers shown in elog:253/1

Database ok!
Stats ok!
ucesb ok!

Temperatures ok! elog:253/2
Statistics ok! elog:253/3
Leakage currents ok! Steadily rising, day/night effect? - elog:253/4 and elog:253/5

16:21

Online histograms zeroed in MIDAS. Same done in MBS DAQ.

16:45

Database ok!
Stats ok!
ucesb ok!

System wide checks:
Clock ok!
ADC 1 failed:
FEE64 module aida06 failed
WR ok!
FPGA ok!
Memory ok!

17:15

Database ok!
Stats ok!
ucesb ok!

17:45

Database ok!
Stats ok!
ucesb ok!

18:15

Database ok!
Stats ok!
ucesb ok!

System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!

MIDAS Rates ok! elog:253/6
Temperatures ok! elog:253/7
Leakage currents ok, but still slowly rising! Topped out at ~20:30 yesterday elog:253/8


18:42
Message in the Merger terminal window (elog:253/9)
MERGE Actor (15671): Working with 0 from 12 data sources


19:05

Database ok!
Stats ok!
ucesb ok!

19:30

Some MIDAS spectra in elog:253/10 and elog:253/11

Database ok!
Stats ok!
ucesb ok!

19:50

Database ok!
Stats ok!
ucesb ok! elog:253/12

System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!

MIDAS Rates ok! elog:253/13
Temperatures ok! elog:253/14
Leakage currents ok, now plateaued! elog:253/15


19:55 - Handing over to James Smile
Attachment 1: Screenshot_2021-04-20_at_15.09.52.png
Screenshot_2021-04-20_at_15.09.52.png
Attachment 2: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-20_Statistics_aidas-gsi.png
Screenshot_2021-04-20_Statistics_aidas-gsi.png
Attachment 4: lpg1.png
lpg1.png
Attachment 5: Screenshot_2021-04-20_at_15.13.14.png
Screenshot_2021-04-20_at_15.13.14.png
Attachment 6: Screenshot_2021-04-20_Statistics_aidas-gsi.png
Screenshot_2021-04-20_Statistics_aidas-gsi.png
Attachment 7: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 8: lpg2.png
lpg2.png
Attachment 9: lpg3_merger.png
lpg3_merger.png
Attachment 10: Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi.png
Attachment 11: Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi(1).png
Attachment 12: Screenshot_2021-04-20_at_18.50.56.png
Screenshot_2021-04-20_at_18.50.56.png
Attachment 13: Screenshot_2021-04-20_Statistics_aidas-gsi(2).png
Screenshot_2021-04-20_Statistics_aidas-gsi(2).png
Attachment 14: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 15: lpg3.png
lpg3.png
  252   Tue Apr 20 10:54:41 2021 CBApril 20 12:00 - 16:00 shift
11:55 CA got AIDA back online

12:00 Stats OK (attach 1)
No faults found
UCESB OK - no beam atm (attach 2)
Temps OK (running hotter than yesterday, but not worrying yet) - attach 3
Bias OK (attach 4)

System wide checks
Clock - *all* pass
ADC - 06 fails
WR - *all* pass
FPGA *all* pass

Memory info
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      4      1      3      2      1      2      4      2      3      7   : 39876
aida02 :     23      6      1      2      2      2      1      4      2      3      7   : 39772
aida03 :     24      4      4      1      2      3      1      4      2      3      7   : 39904
aida04 :     10      3      1      1      1      3      1      4      2      3      7   : 39728
aida05 :     12      2      3      2      1      4      2      3      1      4      7   : 40688
aida06 :     10      5      4      2      2      3      3      4      2      3      7   : 40400
aida07 :     22      6      1      3      1      3      1      4      2      3      7   : 39864
aida08 :     20      4      4      1      2      3      2      4      2      3      7   : 40144
aida09 :      1      2      2      3      1      2      1      3      3      3      7   : 40148
aida10 :      3      3      1      2      2      2      2      3      2      3      7   : 39412
aida11 :     25      7      0      1      0      3      2      3      2      3      7   : 39484
aida12 :     21      7      3      1      3      2      2      3      2      3      7   : 39580


Waiting for beam.

12:42 Stats OK, UCESB ok, No faults. No beam.
13:21 Stats OK, UCESB ok, No faults. No beam.
13:24 Implants in DSSD#3. ASIC check. Disappeared. No beam.

14:00 No beam.
Stats OK (attach 5)
Temps OK (attach 6)
UCESB OK
Bias  OK (attach 7)

System wide checks
Clock OK
ADC calibration 06 fails
WR OK
FPGA OK

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      1      3      2      3      3      2      3      7   : 39848
aida02 :     15      8      2      2      2      3      2      3      2      3      7   : 39644
aida03 :      8      1      2      3      4      3      2      3      2      3      7   : 39720
aida04 :     25     12      3      1      1      3      1      4      2      3      7   : 39892
aida05 :     13      4      4      4      0      3      2      3      1      4      7   : 40596
aida06 :     21      8      3      3      0      3      3      4      2      3      7   : 40356
aida07 :     17     11      0      1      1      3      1      4      2      3      7   : 39804
aida08 :     16      8      2      3      2      3      1      4      2      3      7   : 39936
aida09 :     17      6      3      2      1      2      1      4      2      3      7   : 39716
aida10 :      3      3      2      2      4      1      2      2      3      3      7   : 39940
aida11 :      4      4      4      1      1      3      3      2      2      3      7   : 39248
aida12 :      2      4      1      3      1      2      2      3      2      3      7   : 39384

14:30 Stats OK, UCESB ok, No faults. No beam.
15:00 Stats OK, UCESB ok, No faults. No beam.


15:24 Beam is back!
Stats OK (attach 8), UCESB ok (attach 9), No faults.

15:50 Shift handed over to Liam
Attachment 1: Stats1.png
Stats1.png
Attachment 2: ucesb_nobeam.png
ucesb_nobeam.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temps2.png
Temps2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: UCESB2.png
UCESB2.png
  251   Tue Apr 20 07:06:20 2021 CAApril 20th 08:00 - 12:00
08:09 DAQ crashes at start of shift (!)

      again happens at same time as beam being lost

      All FEE64 lost connection

      Beam downtime for next 3-4 hours

      Performed powercycle and reset

08:40 AIDA back up and running, writing to R36

08:42 *all* system wide checks ok

      temperatures ok - attachment 1

      statistics ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:57 options file size check;


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue Apr 20 08:22:14 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

08:59 no error messages in database check terminal
      stats ok

09:16 waveforms - attachments 4 & 5

09:30 no error messages in database check terminal
      stats ok

10:07 no error messages in database check terminal
      stats ok

10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes

      powercycle attempted, but FEE modules remain unconnected

      TD performs further power cycle

11.12 reboot aida05

      DAQ reset/setup

      system wide checks

      sync asic clocks OK
      clock status & fpga timestamp errors OK
      
      WR decoder status

		 Base 		Current 	Difference
aida09 fault 	 0xb307 : 	 0xb307 : 	 0  
aida09 : WR status 0x40
 aida10 fault 	 0x3d6c : 	 0x3d6c : 	 0  
aida10 : WR status 0x40
 aida11 fault 	 0xa610 : 	 0xa610 : 	 0  
aida11 : WR status 0x40
 aida12 fault 	 0x7c7 : 	 0x7c7 : 	 0  
aida12 : WR status 0x40
 White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)

earlier on, Akash secured HDMI cable to aida07

11:55 AIDA powercycle and reset, DAQ ok now

      writing to file NULL/R45

12:00 CB takes over
Attachment 1: Screenshot_from_2021-04-20_07-45-09.png
Screenshot_from_2021-04-20_07-45-09.png
Attachment 2: Screenshot_from_2021-04-20_07-45-43.png
Screenshot_from_2021-04-20_07-45-43.png
Attachment 3: Screenshot_from_2021-04-20_07-46-05.png
Screenshot_from_2021-04-20_07-46-05.png
Attachment 4: Screenshot_from_2021-04-20_08-14-58.png
Screenshot_from_2021-04-20_08-14-58.png
Attachment 5: Screenshot_from_2021-04-20_08-15-38.png
Screenshot_from_2021-04-20_08-15-38.png
  250   Mon Apr 19 23:01:02 2021 OH20th April 00:00-08:00
23:52 Noticed AIDA07 has dropped out of the merger and is producing no statistics
      Powercycling to recover

00:14 All FEEs recovered from powercycle
      aida09 running again at HEC threshold of 0x2 (Was set to 0x1 earlier in the day while testing the HEC event disparity in DSSD3)
      Now writing to R34
      Following the powercycle the rates in the FEEs are much improved and back to where they were before the 08:30 crash on the 19th. - Attachment 1
      Waveforms also look improved - attachmens 2 and 3


      Current system parameters
      ASIC settings Dec19 (All LEC comparator 0xa)
      Pulser at 2V and 2Hz
      DSSD 1 and 2 Bias 160V
      DSSD 3 at 120V (Will lower to 100 again during the beam off period tomorrow)

     

00:28 In analysis of R34_13 the number of HEC items in DSSD3 appears to make sense again. The number in aida09 are consistent with the number in aida05 and the same for aida11 and aida07.
      This appears to confirm that the earlier HEC issues we were observing were hardware related but we are unsure what/why/how these issues were caused.
      Analysis - attachment 4

00:40 Issue with one of the FRS magnets so they stop recording data. Leaving AIDA running.

01:08 While beam was down have reset DSSD3 bias to 100V as was agreed with Tom and Nic in email
      Leakage current decreased as expected - attachment 5
      Once beam is back will check HEC still as expected.

01:14 Last magnet in FRS has an issue with the water flowing. Either sensor issue or water flow is to low.
      Hakke have increased the water flow to hopefully keep the water running.
      Beam is back.

      From ucesb it seems that the HEC events in DSSD3 are still appearing as expected. Will check a file once we have a complete file.
      Checked R34_36 and the HEC events in DSSD3 are still consistent - attachment 6

2:20 Statistics ok - attachment 7
     Temp - attachment 8
     Bias and leakage current ok - attachment 9

     All system wide checks ok except for WR error check:
	
		 Base 		Current 	Difference
aida07 fault 	 0x9154 : 	 0x915b : 	 7  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Note aida07 reports two bad timestamps in the merger
     MERGE Data Link (3762): bad timestamp  6 3 0xc1817ea7 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
     MERGE Data Link (3762): bad timestamp  6 3 0xc1b77e25 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56


02:25 Remembered during a pre-experiment meeting with Vic, Patrick and Carl they mentioned the merger would track timestamp sequence errors
      Can confirm this is true. Can see the two in aida07 in the counter in the merger stats page - attachment 10

02:45 Now up to 7 timestamp errors on aida07 in the merger
      
02:49 Ran R34_45 through AIDASort - It reports a greater number of implant events are able to have a front back match performed
      Ratio between aida07 and 09 is 0.71 -> Double what it was before
      Energy response of the HEC channels in aida07 and aida09 looks equivalent (left group aida09 right group aida11) - attahchment 11 

04:40 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents - attachment 14
      Clock status, ADC calibration and memory infomration all ok
      aida07 fails both WR and FPGA
	
		 Base 		Current 	Difference
aida07 fault 	 0x915b : 	 0x9161 : 	 6  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


       TS errors in the merger still only at 7 with all in aida07

06:25 Statistics - attachment 15
      Temps ok - attachment 16
      Bias and leakage currents ok - attachment 17

      System wide checks: clock status, adc calibration and memory information all ok
      WR multiple fails
	
		 Base 		Current 	Difference
aida05 fault 	 0xc9ca : 	 0xc9cb : 	 1  
aida06 fault 	 0xe900 : 	 0xe901 : 	 1  
aida07 fault 	 0x915b : 	 0x9166 : 	 11  
aida08 fault 	 0x641e : 	 0x641f : 	 1  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     FPGA aida07 fails with difference of 1

     No new TS sequence errors in merger      
      

07:10 Beam has dropped out.
07:14 Beam is back
Attachment 1: 210420_0016_Stats.png
210420_0016_Stats.png
Attachment 2: 210420_0021_Layout7.png
210420_0021_Layout7.png
Attachment 3: 210420_0022_Layout8.png
210420_0022_Layout8.png
Attachment 4: 210420_0026_R34_13_Analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255975543 ( 1356743.5 Hz)
        Other data format:    5944457 (   31507.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32703 (     173.3 Hz)
                              WR48-63:      32703 (     173.3 Hz)
                           FEE64 disc:    1073243 (    5688.5 Hz)
                             MBS info:    4805808 (   25472.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     345437 (    1830.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      188.669 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               46.534
  2                0.000               63.789
  3                0.000               54.847
  4                0.000              174.519
  5                0.000              123.945
  6                0.000              113.898
  7                0.000              135.923
  8                0.000               59.774
  9                0.000               27.296
 10                0.000               12.263
 11                0.000               65.598
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14400003     100475          0          0          0          0       1767       1767      95813       1128          0      35602
  1   23251868      63483          0          0          0          0       2988       2988      57507          0          0      21204
  2   17192461    2025458          0          0          0          0       2415       2415     200396    1820232          0      34093
  3   20019932    1908259          0          0          0          0       2815       2815      81941    1820688          0      32715
  4    3059738     405674          0          0          0          0        403        403     100680     304188          0      35774
  5   10595698     355900          0          0          0          0       1322       1322      58122     295134          0      21319
  6   12037952     388178          0          0          0          0       1554       1554     105230     279840          0      34521
  7    9390918     368608          0          0          0          0       1205       1205      81600     284598          0      31165
  8   46854628      72459          0          0          0          0       5872       5872      60715          0          0      21758
  9   43874737      60599          0          0          0          0       5426       5426      49747          0          0      17382
 10   36184433     112868          0          0          0          0       4528       4528     103812          0          0      31745
 11   19113175      82496          0          0          0          0       2408       2408      77680          0          0      28159
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 1641.051s (   19.500 blocks/s,   1.219 Mb/s)
Attachment 5: 210420_0109_Bias.png
210420_0109_Bias.png
Attachment 6: 210420_0026_R34_36_Analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255999603 ( 1309621.8 Hz)
        Other data format:    5920397 (   30287.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32728 (     167.4 Hz)
                              WR48-63:      32728 (     167.4 Hz)
                           FEE64 disc:    1044504 (    5343.4 Hz)
                             MBS info:    4810437 (   24608.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     327887 (    1677.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      195.476 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               51.325
  2                0.000               68.599
  3                0.000               55.115
  4                0.000              180.174
  5                0.000              140.295
  6                0.000              123.427
  7                0.000              138.446
  8                0.000               55.510
  9                0.000               13.463
 10                0.000               24.357
 11                0.000               74.336
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14611253      94841          0          0          0          0       1936       1936      89802       1167          0      33511
  1   23423897      61346          0          0          0          0       3006       3006      55334          0          0      20222
  2   17311487    2050104          0          0          0          0       2477       2477     195362    1849788          0      32021
  3   20533060    1944396          0          0          0          0       2862       2862      77493    1861179          0      30741
  4    2942569     381933          0          0          0          0        417        417      93930     287169          0      33667
  5    9776355     338395          0          0          0          0       1239       1239      56110     279807          0      20096
  6   11887130     369619          0          0          0          0       1555       1555     103862     262647          0      32503
  7    9239609     348065          0          0          0          0       1200       1200      76985     268680          0      29470
  8   49507336      72711          0          0          0          0       6129       6129      60453          0          0      21190
  9   44170393      65100          0          0          0          0       5430       5430      54240          0          0      18016
 10   32977762     112783          0          0          0          0       4022       4022     104739          0          0      30280
 11   19618752      81104          0          0          0          0       2455       2455      76194          0          0      26170
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 5995.524s (    5.337 blocks/s,   0.334 Mb/s)
Attachment 7: 210420_0220_Stats.png
210420_0220_Stats.png
Attachment 8: 210420_0229_Temp.png
210420_0229_Temp.png
Attachment 9: 210420_0220_Bias.png
210420_0220_Bias.png
Attachment 10: 210420_0225_MergerStats.png
210420_0225_MergerStats.png
Attachment 11: Capture.PNG
Capture.PNG
Attachment 12: 210420_0437_stats.png
210420_0437_stats.png
Attachment 13: 210420_0538_Temp.png
210420_0538_Temp.png
Attachment 14: 210420_0439_Bias.png
210420_0439_Bias.png
Attachment 15: 210420_0624_stats.png
210420_0624_stats.png
Attachment 16: 210420_0624_Temp.png
210420_0624_Temp.png
Attachment 17: 210420_0625_Bias.png
210420_0625_Bias.png
  249   Mon Apr 19 19:19:18 2021 CBMonday 19 April 20:00-24:00 shift
20:19 
Statistics OK (attach 1)
No faults found
Ucesb OK (attach 2)
Temps OK (attach 3)
Bias OK (attach 4)

System-wide checks - as per last check
aida09 fails clock (OK)
aida07 and 09 fail calibration (OK)

		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05f : 	 13  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      0      3      2      2      2      5      2      3      2      3      6   : 35704
aida02 :     19      4      1      1      1      2      2      3      3      3      6   : 36316
aida03 :     12      9      2      1      2      2      2      3      3      3      6   : 36408
aida04 :     16      3      5      4      2      4      2      3      3      3      6   : 36776
aida05 :     41     10     10      3      3      3      3      3      3      3      6   : 37172
aida06 :     25     21      9      3      2      3      1      4      3      3      6   : 37116
aida07 :     10      7      1      2      1      3      2      2      3      3      6   : 35952
aida08 :     33      7      5      1      1      2      2      3      3      3      6   : 36460
aida09 :     12      7      4      2      2      2      2      2      2      4      6   : 36968
aida10 :     16      6      2      1      1      4      3      3      2      3      6   : 35824
aida11 :      2      0      1      2      2      2      2      4      2      3      6   : 35800
aida12 :     15      6      2      0      2      4      2      2      3      3      6   : 36108



20:48 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[...]_*

21:00 Stats OK, UCESB OK, No faults

21:07 Go4 spectra erased by remote shifters.

21:15 Local shifters report correlations lost. Beam may be intermittently removed for checks.

21:30 Still no beam. DAQ issues locally. AIDA MBS link had to be restarted due to operator mistake.




21:37 Beam back. Stats OK, UCESB OK, No faults found.

System wide checks:
Clock - aida09 fails

All modules now fail ADC calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd061 : 	 15  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x36 : 	 54  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      3      1      2      4      2      4      2      3      6   : 36104
aida02 :      2      3      0      2      2      4      2      2      3      3      6   : 36064
aida03 :      6      3      3      3      2      2      2      2      4      3      6   : 36928
aida04 :     15      5      8      4      3      3      2      3      3      3      6   : 36772
aida05 :     10      5      8      4      4      4      3      2      3      3      6   : 36688
aida06 :     14     17     10      3      3      3      1      4      3      3      6   : 37120
aida07 :      3      4      1      1      1      2      2      2      3      3      6   : 35740
aida08 :     12      5      7      2      2      4      1      2      3      3      6   : 35976
aida09 :      0      6      2      0      2      2      2      2      2      4      6   : 36816
aida10 :      4      2      5      2      1      4      2      4      2      3      6   : 36080
aida11 :     14      3      2      3      1      2      2      4      2      3      6   : 35856
aida12 :      1      2      0      2      1      4      1      3      3      3      6   : 36244

22:05 Stats OK (attach 5)
Temps OK (attach 6)
Bias OK (attach 7)

System wide checks as before - see above.

22:13 ADC calibration - Only 06,07 and 09 now fail
(TD re-calibrated remotely)

22:53 Beam off to allow access to S4 to reboot VME crate (akash).


22:58 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[1-2][0-9]_* R[3][0-2]_*
aka everything *except* R33_*


23:11: Beam is back. Stats and UCESB as before.

23:24: Stats OK (attach 8)
Temps OK (attach 9)
Bias OK (attach 10)
UCESB OK (attach 11)
System-wide checks
Clock: 06 fails
Calibration: 06,07,09 fail

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      2      2      3      4      3      3      2      3      6   : 35996
aida02 :      8     12      4      1      2      4      3      2      3      3      6   : 36448
aida03 :      2      4      2      4      3      3      1      3      3      3      6   : 36360
aida04 :      7      8      4      4      2      4      3      3      3      3      6   : 37020
aida05 :      3      5      7      4      3      3      3      3      3      3      6   : 36964
aida06 :      2      6      8      2      3      4      2      3      3      3      6   : 36792
aida07 :      8      4      5      2      0      3      1      2      3      4      6   : 37712
aida08 :     26     11      3      1      1      3      1      3      3      3      6   : 36304
aida09 :     22      4      2      2      3      4      1      3      3      3      6   : 36504
aida10 :     21      6      6      2      0      4      2      3      3      3      6   : 36644
aida11 :     22      7      2      3      2      1      2      4      2      3      6   : 35856
aida12 :     16      9      2      2      3      3      2      2      3      3      6   : 36136
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Ucesb1.png
Ucesb1.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temp2.png
Temp2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: Temps3.png
Temps3.png
Attachment 10: Bias3.png
Bias3.png
Attachment 11: Ucesb2.png
Ucesb2.png
  248   Mon Apr 19 16:58:29 2021 LJWMonday 19th April 16:00-20:00 Shift
System Checks @ ~18:10 :


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd05a : 	 8  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x23 : 	 35  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      6      4      0      1      5      2      3      2      3      6   : 35636
aida02 :     27      3      2      0      2      3      2      3      3      3      6   : 36516
aida03 :     33      9      2      1      1      3      2      3      3      3      6   : 36556
aida04 :     11      6      5      4      2      4      3      3      3      3      6   : 37036
aida05 :     15      8      7      4      3      4      4      2      3      3      6   : 36908
aida06 :     13      9      7      3      3      4      2      3      3      3      6   : 36876
aida07 :      1      3      3      2      2      3      1      3      3      3      6   : 36236
aida08 :     29      5      2      1      1      2      2      3      3      3      6   : 36380
aida09 :      3      8      2      0      2      3      2      2      2      4      6   : 36972
aida10 :      9      5      3      3      1      4      2      4      2      3      6   : 36124
aida11 :      5      7      3      2      2      2      2      4      2      3      6   : 35900
aida12 :     24      6      3      1      2      3      2      2      3      3      6   : 36064


System Checks @ ~19:55

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05e : 	 12  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x27 : 	 39  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18     10      2      2      1      5      2      3      2      3      6   : 35768
aida02 :     21      9      2      1      2      2      2      3      3      3      6   : 36444
aida03 :     13      5      6      1      2      2      2      3      3      3      6   : 36444
aida04 :     30     11      4      3      2      4      3      3      3      3      6   : 37104
aida05 :     10      9      8      4      4      4      4      2      3      3      6   : 36976
aida06 :     22     17     10      3      2      3      1      4      3      3      6   : 37088
aida07 :     20      8      4      1      2      4      1      2      3      3      6   : 35952
aida08 :     14      4      3      0      1      2      2      3      3      3      6   : 36296
aida09 :     24      8      4      0      1      3      3      3      3      3      6   : 36768
aida10 :     19      8      4      1      1      4      2      3      3      3      6   : 36652
aida11 :      6      6      1      2      3      1      2      4      2      3      6   : 35800
aida12 :     15      7      1      1      2      4      1      2      3      3      6   : 35876
Attachment 1: 63.png
63.png
Attachment 2: Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Attachment 3: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 4: 64.png
64.png
Attachment 5: 65.png
65.png
Attachment 6: 66.png
66.png
  246   Mon Apr 19 13:27:08 2021 RDPMonday 19th April 12 noon shift
12.30 There appear to be fewer HEC events in aida09 cf. aida11 and upstream DSSSDs
      Decrease aida09 asic #1-4 HEC fast comparator 0x2 -> 0x1
      Online/nearline analyses of daat files indicates that this does not resolve the issue

Some timestamp issues 

MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0287be94 0x00003dd78287be94 0x16773dd78287be94 0x16773dd78287beb2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610040 0x073302ce 0x00003df3e73302ce 0x16773df3e73302ce 0x16773df3e73302ec
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x0a54b794 0x00003e4e4a54b794 0x16773e4e4a54b794 0x16773e4e4a54b7b2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x052da21e 0x00003f68f52da21e 0x16773f68f52da21e 0x16773f68f52da232
MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0061489e 0x000040aa8061489e 0x167740aa8061489e 0x167740aa806148a8
MERGE Data Link (23725): bad timestamp  11 3 0x8b610008 0x04029d5e 0x00004169c4029d5e 0x16774169c4029d5e 0x16774169c4029d68
MERGE Data Link (23725): bad timestamp  11 3 0x8b630004 0x08dfcb14 0x0000417fd8dfcb14 0x1677417fd8dfcb14 0x1677417fd8dfcb28
MERGE Data Link (23725): bad timestamp  11 3 0x8b630020 0x08dfcb1e 0x0000417fd8dfcb1e 0x1677417fd8dfcb1e 0x1677417fd8dfcb28

13.50 Increased DSSSD# 3 bias from -100V

      -100V -6.325uA
      -110V -6.500uA
      -120V -6.800uA



       
      
Attachment 1: 61.png
61.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 3: Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
  244   Mon Apr 19 06:46:15 2021 MSMonday 19 April 07:47-12:00 2021

07:47

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c46 :      74  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

        
             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x35 :      43  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     18     10      7      1      3      2      4      2      3      6   : 36380
aida02 :     11     15      5      3      2      3      2      3      1      4      6   : 36692
aida03 :     15     11     13      3      4      3      3      3      2      3      6   : 36164
aida04 :     22     14     20      5      1      3      4      3      2      3      6   : 36456
aida05 :      7      7      5      2      1      2      3      4      2      3      6   : 36132
aida06 :     21     14      6      2      3      3      1      2      2      4      6   : 37028
aida07 :     11      8      7      4      2      3      2      2      4      3      6   : 37212
aida08 :     18     10      7      5      3      1      4      3      2      3      6   : 36072
aida09 :     24     23     18      3      2      3      2      4      2      3      6   : 36504
aida10 :      6     19      5      2      2      2      2      3      2      3      6   : 35520
aida11 :     22     15      7      2      1      3      2      3      2      3      6   : 35648
aida12 :     35     13      8      2      5      3      1      3      3      3      6   : 36724

 

At 8:30 DAQ and Temperature Scan stopped working properly.

 

 

11:00

It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence

TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.

Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.

In the waveforms large 100kHz transiets could be seen - attachment 4

A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.

The waveforms here are much improved though - attachment 5 and 6

 

 

11:45

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      1      2      2      1      2      4      2      4      2      3      7   : 40148
aida02 :      5      4      2      2      1      2      2      3      2      3      7   : 39380
aida03 :      3      2      1      1      1      3      1      4      2      3      7   : 39692
aida04 :      4      2      1      0      1      3      3      3      2      3      7   : 39664
aida05 :      2      7      2      0      2      2      3      4      2      3      7   : 40160
aida06 :      2      3      2      2      2      2      1      3      3      3      7   : 40192
aida07 :     23      7      2      0      0      3      2      4      2      3      7   : 39988
aida08 :     23     11      4      0      1      3      1      4      2      3      7   : 39860
aida09 :      1      3      1      1      2      4      1      3      1      4      7   : 40396
aida10 :      1      3      3      1      1      4      1      2      3      3      7   : 39852
aida11 :      3      0      2      3      1      2      3      2      2      3      7   : 39116
aida12 :      5      8      0      3      1      4      2      3      2      3      7   : 39668

 

 

 

 

 

 

 

 

Attachment 1: Screenshot_2021-04-19_Statistics_aidas-gsi.png
Screenshot_2021-04-19_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 42.png
42.png
Attachment 4: Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: 210319_1054_Layout7.png
210319_1054_Layout7.png
Attachment 6: 210419_1054_Layout8.png
210419_1054_Layout8.png
Attachment 7: Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 43.png
43.png
  243   Mon Apr 19 01:08:40 2021 MAMonday 19th April 00:00-08:00

03:09   General check

Rates, temptature, voltages are OK attached 1, 2, 3, 4

 

****Clock Ckeck******

OK
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

****** White Rabbit check******

     Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c2c :      48  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

*****FPGA check ******


             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x10 :      6  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

*****Memory check*****


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6     12     11      7      2      3      2      4      2      3      6   : 36360
aida02 :     21     15      7      3      2      3      1      4      2      3      6   : 35996
aida03 :      9     10     10      2      4      3      3      3      2      3      6   : 36052
aida04 :     17     17     17      4      0      2      3      4      2      3      6   : 36444
aida05 :     25      7      3      1      1      1      2      3      2      4      6   : 37292
aida06 :     29     15      7      2      3      4      1      3      3      3      6   : 36700
aida07 :     17     17      6      5      2      3      2      3      3      3      6   : 36812
aida08 :     22      6     10      5      3      1      3      4      2      3      6   : 36360
aida09 :     18     28     14      4      2      3      3      3      2      3      6   : 36232
aida10 :     26     16      4      3      3      1      3      4      2      3      6   : 36296
aida11 :     16     10      4      0      3      3      1      4      2      3      6   : 35856
aida12 :     19     14      4      2      4      4      1      3      3      3      6   : 36668

 

04:15      General Check

Rates, Tempratures, Voltages are ok, attached 5,6,7,8

******Clock Check*******

OK

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******ADC******

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******White Rabbit check******


         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c39 :      61  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


******FPGA Check******

     Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x11 :      7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


****** Memorey check******


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     21     10      7      1      3      2      4      2      3      6   : 36436
aida02 :     28     21      3      3      1      3      1      4      2      3      6   : 35944
aida03 :     15      8     11      3      3      3      3      3      2      3      6   : 36044
aida04 :     25     19     14      4      1      3      4      3      2      3      6   : 36380
aida05 :     12      6      5      1      0      2      2      3      2      4      6   : 37328
aida06 :     29     21      4      2      3      4      1      3      3      3      6   : 36700
aida07 :     19     14      7      6      2      3      2      2      3      3      6   : 36332
aida08 :     11      8      8      6      3      1      3      4      2      3      6   : 36332
aida09 :     16     25     11      3      1      2      2      4      2      3      6   : 36184
aida10 :     14     13      4      2      3      2      2      4      2      3      6   : 36064
aida11 :      4      9      4      0      3      3      1      4      2      3      6   : 35800
aida12 :     38     17      4      2      4      4      1      3      3      3      6   : 36768

 

 

 

 

Attachment 1: 210419_0302_Voltages.png
210419_0302_Voltages.png
Attachment 2: 210419_0303_Stats.png
210419_0303_Stats.png
Attachment 3: 210419_0303_Temp.png
210419_0303_Temp.png
Attachment 4: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 5: 210419_0402_Stats.png
210419_0402_Stats.png
Attachment 6: 210419_0402_Voltages.png
210419_0402_Voltages.png
Attachment 7: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 8: 210419_0414_ucesb.png
210419_0414_ucesb.png
  242   Sun Apr 18 21:43:01 2021 ML-OHsystem wide checks

New System wide check 10pm CET

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration check:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c1d :      33  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  


White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory from FEE64:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22     19      6      4      2      3      2      4      2      3      6   : 36304
aida02 :     23      9      3      3      1      3      1      4      2      3      6   : 35828
aida03 :     15      5      6      3      2      3      3      3      2      3      6   : 35876
aida04 :     20      8     18      1      1      3      4      3      2      3      6   : 36240
aida05 :     21      7      1      2      1      2      4      3      2      3      6   : 35868
aida06 :     18      8      4      1      3      4      1      3      3      3      6   : 36520
aida07 :      4      9      3      4      2      3      2      2      3      3      6   : 36104
aida08 :     15      5      2      4      2      1      3      4      2      3      6   : 36100
aida09 :      2     23      8      1      2      3      2      4      2      3      6   : 36192
aida10 :     10      7      8      5      2      1      3      3      2      3      6   : 35712
aida11 :      1      9      5      1      2      3      1      4      2      3      6   : 35772
aida12 :     14      8      6      3      3      4      1      3      3      3      6   : 36600

Attachment 1: 18-04-2021_10pm_HV.png
18-04-2021_10pm_HV.png
Attachment 2: 18-04-2021_10pm_StatGoodEvt.png
18-04-2021_10pm_StatGoodEvt.png
Attachment 3: 18-04-2021_10pm_PAUSE.png
18-04-2021_10pm_PAUSE.png
Attachment 4: 18-04-2021_10pm_RESUME.png
18-04-2021_10pm_RESUME.png
Attachment 5: 18-04-2021_10pm_ADCdataItem.png
18-04-2021_10pm_ADCdataItem.png
Attachment 6: 18-04-2021_10pm_DISC.png
18-04-2021_10pm_DISC.png
Attachment 7: 18-04-2021_10pm_correlations.png
18-04-2021_10pm_correlations.png
Attachment 8: 18-04-2021_10pm_SpecRate.png
18-04-2021_10pm_SpecRate.png
Attachment 9: 18-04-2021_10pm_Spec1-8H.png
18-04-2021_10pm_Spec1-8H.png
Attachment 10: 18-04-2021_10pm_Spec1-8HlayoutID6.png
18-04-2021_10pm_Spec1-8HlayoutID6.png
Attachment 11: 18-04-2021_10pm_Spec1-8L.png
18-04-2021_10pm_Spec1-8L.png
Attachment 12: 18-04-2021_10pm_Spec1-8L_layoutid4.png
18-04-2021_10pm_Spec1-8L_layoutid4.png
Attachment 13: 18-04-2021_10pm_Spec1-8W.png
18-04-2021_10pm_Spec1-8W.png
Attachment 14: 18-04-2021_10pm_SpecLayoutID8.png
18-04-2021_10pm_SpecLayoutID8.png
Attachment 15: 18-04-2021_10pm_WR28-47.png
18-04-2021_10pm_WR28-47.png
Attachment 16: 18-04-2021_10pm_WR48-63.png
18-04-2021_10pm_WR48-63.png
Attachment 17: 18-04-2021_10pm_temp.png
18-04-2021_10pm_temp.png
Attachment 18: 18-04-2021_10pm_ucesb_monitor.png
18-04-2021_10pm_ucesb_monitor.png
Attachment 19: 18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
Attachment 20: 18-04-2021_10pm_NewMerger.png
18-04-2021_10pm_NewMerger.png
  241   Sun Apr 18 19:05:33 2021 ML-OHAll spectra reset

 All spectra reset at 8pm CET.

 

  240   Sun Apr 18 17:27:44 2021 ML-OHsystem wide checks

system wide checks at 6pm CET

##################################################

Difference noted from previous shift:

FPGS timestamp error 10 passed and 2 failed

##################################################


Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit checks:

         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c11 :      21  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA timestamp errors checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xc :      2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory information fro FEE64:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     16      4      2      2      2      1      3      3      3      6   : 36332
aida02 :     19      8      9      3      2      2      2      4      2      3      6   : 36092
aida03 :     23     10      8      6      4      4      3      3      2      3      6   : 36332
aida04 :     27      4      8      0      1      3      4      3      2      3      6   : 36044
aida05 :     21      6      5      4      4      2      2      2      2      4      6   : 37204
aida06 :     23     11      5      0      3      4      1      3      3      3      6   : 36548
aida07 :     14      9      5      2      3      3      2      2      4      3      6   : 37200
aida08 :     26      8      7      2      2      1      4      3      2      3      6   : 35928
aida09 :      2      3      4      1      1      3      3      4      2      3      6   : 36160
aida10 :     16      9      2      5      1      1      3      4      2      3      6   : 36104
aida11 :      7      5      0      0      3      3      1      4      2      3      6   : 35716
aida12 :      9      7      3      2      4      4      1      3      3      3      6   : 36556

 

 

 

Attachment 1: 18-04-2021_6pm_CAENHV.png
18-04-2021_6pm_CAENHV.png
Attachment 2: 18-04-2021_6pm_StatGoodEvent.png
18-04-2021_6pm_StatGoodEvent.png
Attachment 3: 18-04-2021_6pm_ADCdataItem.png
18-04-2021_6pm_ADCdataItem.png
Attachment 4: 18-04-2021_6pm_WR28-47#4.png
18-04-2021_6pm_WR28-47#4.png
Attachment 5: 18-04-2021_6pm_WR48-63#5.png
18-04-2021_6pm_WR48-63#5.png
Attachment 6: 18-04-2021_6pm_PauseInfo.png
18-04-2021_6pm_PauseInfo.png
Attachment 7: 18-04-2021_6pm_ResumeInfo.png
18-04-2021_6pm_ResumeInfo.png
Attachment 8: 18-04-2021_6pm_DiscInfo.png
18-04-2021_6pm_DiscInfo.png
Attachment 9: 18-04-2021_6pm_Correlation#8.png
18-04-2021_6pm_Correlation#8.png
Attachment 10: 18-04-2021_6pm_ucesb.png
18-04-2021_6pm_ucesb.png
Attachment 11: 18-04-2021_645pm_tempScan.png
18-04-2021_645pm_tempScan.png
Attachment 12: 18-04-2021_645pm_Spec1-8L.png
18-04-2021_645pm_Spec1-8L.png
Attachment 13: 18-04-2021_7pm_Spec1-8L_2.png
18-04-2021_7pm_Spec1-8L_2.png
Attachment 14: 18-04-2021_7pm_Spec1-8_layoutId4.png
18-04-2021_7pm_Spec1-8_layoutId4.png
Attachment 15: 18-04-2021_7pm_Spec1-8H.png
18-04-2021_7pm_Spec1-8H.png
Attachment 16: 18-04-2021_645pm_SpecRate.png
18-04-2021_645pm_SpecRate.png
Attachment 17: 18-04-2021_730pm_AIDA_LC.png
18-04-2021_730pm_AIDA_LC.png
  239   Sun Apr 18 11:18:21 2021 TDSunday 18 April 12.00-16.00
12.18 DAQ continues file NULL/R30_490

      ASIC settings 2019Dec19-16.19.51
      slow comparator 0x64 -> 0xa

      BNC PB-5
      amplitude 2V
      attenuation 1x
      decay time 1ms
      frequency 2Hz

12.22 all histograms zero'd

c. 12.30 219Rn setting

12.35 analysis of R30_490 - see attachment 1

13.15 system wide checks

Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida01 fault 	 0x7686 : 	 0x768a : 	 4  
aida02 fault 	 0x941d : 	 0x9421 : 	 4  
aida03 fault 	 0x7cd7 : 	 0x7cdb : 	 4  
aida04 fault 	 0xb86d : 	 0xb871 : 	 4  
aida05 fault 	 0x1a59 : 	 0x1a5f : 	 6  
aida06 fault 	 0x4f45 : 	 0x4f4b : 	 6  
aida07 fault 	 0x3bfc : 	 0x3c04 : 	 8  
aida08 fault 	 0xc7ce : 	 0xc7d4 : 	 6  
aida09 fault 	 0xb33b : 	 0xb33c : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0xa : 	 0xb : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :      2     11      7      4      2      2      2      4      2      3      6   : 36048
aida03 :     21      8      6      4      5      4      3      3      2      3      6   : 36276
aida04 :     19     11     14      3      2      3      2      4      2      3      6   : 36324
aida05 :     25      4      5      4      4      2      1      3      2      4      6   : 37460
aida06 :     13      9     15      1      3      5      1      3      3      3      6   : 36812
aida07 :     19     10      6      1      3      3      2      2      4      3      6   : 37212
aida08 :     23      6      4      2      1      1      3      4      2      3      6   : 36044
aida09 :     11      5      5      1      2      4      3      3      2      3      6   : 35908
aida10 :     14      5      6      5      3      1      3      3      2      3      6   : 35744
aida11 :     16      5      3      0      3      3      1      4      2      3      6   : 35800
aida12 :     12     10      0      2      4      4      1      3      3      3      6   : 36544

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

*no faults detected by ~/oh/OptionsCheck.py

13.10 Detector biases & leakage currents OK - see attachment 2
      Statistics - good events, adc data, disc data, info code 4 & 5, pause, resume, correlation info - attachments 3-12
      per FEE64 rate spectra - attachments 13-14
      1.8.L spectra - attachments 15-18

       pulser peak widths
       1 80
       2 113
       3 308
       4 105
       5 50
       6 91
       7 61
       8 90
       9 200
       10 133
       11 187
       12 107
  
      1.8.H spectra - attachments 19-20
      1.8.W spectra - attachments 21-22
      ucesb - attachment 23
       implant rates DSSSD#1-3 to c. 1kHz
      grafana - AIDA DSSSD leakage currents for previous 2 days - attachment 24
      merger & merger/tape server/mbs data tranfser - attachment 25
       no recent merger warning/error messages

13.30 225At setting
Attachment 1: R30_490
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256516272 ( 1223332.2 Hz)
        Other data format:    5403728 (   25770.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32781 (     156.3 Hz)
                              WR48-63:      32781 (     156.3 Hz)
                           FEE64 disc:     828389 (    3950.6 Hz)
                             MBS info:    4509777 (   21507.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     263493 (    1256.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      209.687 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               76.433
  2                0.000               87.604
  3                0.000               78.795
  4                0.000              195.707
  5                0.000              158.854
  6                0.000              193.755
  7                0.000              152.656
  8                0.000               23.126
  9                0.000               23.243
 10                0.000               11.178
 11                0.000              108.457
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9851554      72944          0          0          0          0       1261       1261      69168       1254          0      26448
  1   21158309      48956          0          0          0          0       2701       2701      43554          0          0      16374
  2   17027137    1992411          0          0          0          0       2376       2376     156273    1831386          0      25239
  3   18790255    1894647          0          0          0          0       2567       2567      58490    1831023          0      23746
  4    2269151     296174          0          0          0          0        339        339      74660     220836          0      26646
  5    8373135     260125          0          0          0          0       1152       1152      43783     214038          0      16230
  6    4256540     286725          0          0          0          0        532        532      81589     204072          0      25586
  7    9526319     267328          0          0          0          0       1246       1246      57668     207168          0      22643
  8   54059298      76554          0          0          0          0       6619       6619      63316          0          0      22746
  9   49446691      53505          0          0          0          0       6256       6256      40993          0          0      14045
 10   45199495      92635          0          0          0          0       5711       5711      81213          0          0      23246
 11   16558388      61724          0          0          0          0       2021       2021      57682          0          0      20544
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:44645.059s (    0.717 blocks/s,   0.045 Mb/s)
Attachment 2: 50.png
50.png
Attachment 3: Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 5: Screenshot_2021-04-18_Statistics_aidas-gsi(2).png
Screenshot_2021-04-18_Statistics_aidas-gsi(2).png
Attachment 6: Screenshot_2021-04-18_Statistics_aidas-gsi(3).png
Screenshot_2021-04-18_Statistics_aidas-gsi(3).png
Attachment 7: Screenshot_2021-04-18_Statistics_aidas-gsi(4).png
Screenshot_2021-04-18_Statistics_aidas-gsi(4).png
Attachment 8: Screenshot_2021-04-18_Statistics_aidas-gsi(5).png
Screenshot_2021-04-18_Statistics_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-18_Statistics_aidas-gsi(6).png
Screenshot_2021-04-18_Statistics_aidas-gsi(6).png
Attachment 10: Screenshot_2021-04-18_Statistics_aidas-gsi(7).png
Screenshot_2021-04-18_Statistics_aidas-gsi(7).png
Attachment 11: Screenshot_2021-04-18_Statistics_aidas-gsi(8).png
Screenshot_2021-04-18_Statistics_aidas-gsi(8).png
Attachment 12: Screenshot_2021-04-18_Statistics_aidas-gsi(9).png
Screenshot_2021-04-18_Statistics_aidas-gsi(9).png
Attachment 13: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(1).png
Attachment 15: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(3).png
Attachment 17: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(4).png
Attachment 18: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(5).png
Attachment 19: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(6).png
Attachment 20: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(7).png
Attachment 21: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(8).png
Attachment 22: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(9).png
Attachment 23: Screenshot_2021-04-18_ucesb.png
Screenshot_2021-04-18_ucesb.png
Attachment 24: Screenshot_2021-04-18_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-18_AIDA_Alerting_-_Grafana.png
Attachment 25: 51.png
51.png
  238   Sun Apr 18 07:30:20 2021 DJ, TDSunday 18th April 08:00-12:00

---- 09-26
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

---


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

--


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

 

08.45 TD resets baseline for WR and FPGA errors

 

Attachment 1: Screenshot_2021-04-18_Statistics_aidas-gsi.png
Screenshot_2021-04-18_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 52.png
52.png
Attachment 4: 53.png
53.png
Attachment 5: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 6: 54.png
54.png
Attachment 7: Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
  237   Sun Apr 18 00:57:36 2021 BA, MASanday 18 April 00.00-08.00

02:01 Beam has stopped at 01:37 and returend at 01:43 for few min and then stopped again and not knowen how it will take until it is back

AIDA scalers attached 1

statistic attached 2

temretuer attached 3

bias attached 4

Clock check ok

ADC check :

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


         Base         Current     Difference
aida05 fault      0x1a52 :      0x1a55 :      3  
aida06 fault      0x4f3e :      0x4f41 :      3  
aida07 fault      0x3bcd :      0x3bea :      29  
aida08 fault      0xc7c7 :      0xc7ca :      3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0x9 :      9  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     32      8      3      1      0      3      1      3      3      3      6   : 36240
aida02 :     11      7      9      3      1      2      2      4      2      3      6   : 35988
aida03 :     32      4     13      5      5      4      3      3      2      3      6   : 36432
aida04 :     26      7      6      1      2      3      2      4      2      3      6   : 36128
aida05 :     18      8      8      7      4      2      2      2      2      4      6   : 37352
aida06 :     24     11      3      1      2      5      1      3      3      3      6   : 36616
aida07 :     14      5      3      3      3      2      3      2      4      3      6   : 37296
aida08 :     21      7      1      5      0      1      2      3      3      3      6   : 36284
aida09 :      0      7      1      1      1      3      2      2      3      3      6   : 35880
aida10 :     22      5      3      4      1      2      2      2      3      3      6   : 35952
aida11 :      4      3      1      1      2      2      3      3      2      3      6   : 35544
aida12 :     22     10      5      4      4      4      1      3      3      3      6   : 36728

 

02:19  beam is back

03:58 The beam has not been stable yet

           The rate reach 1.5 kHz, they will contact FRS team to lower the intensity of the beam

 

AIDA scalers attached 8

statistic attached 7

temretuer attached 6

bias attached 5

 

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

 

 

 

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

   
         Base         Current     Difference
aida05 fault      0x1a52 :      0x1a55 :      3  
aida06 fault      0x4f3e :      0x4f41 :      3  
aida07 fault      0x3bcd :      0x3beb :      30  
aida08 fault      0xc7c7 :      0xc7ca :      3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0x9 :      9  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     21      8      6      2      1      2      1      3      3      3      6   : 36212
aida02 :     24     14     14      2      1      2      2      4      2      3      6   : 36144
aida03 :     31      5     10      5      5      4      2      3      2      3      6   : 36132
aida04 :     12     11     12      2      2      2      3      4      2      3      6   : 36360
aida05 :     23      8      5      7      4      2      2      2      2      4      6   : 37324
aida06 :     21     14     14      1      3      5      1      3      3      3      6   : 36868
aida07 :     15      9      5      0      3      2      3      2      4      3      6   : 37268
aida08 :     21     11     10      3      0      1      2      3      3      3      6   : 36396
aida09 :      5      7      5      1      1      3      1      3      3      3      6   : 36220
aida10 :     15     13     12      3      2      1      2      2      3      3      6   : 36036
aida11 :     13     10      1      0      2      2      2      4      2      3      6   : 35860
aida12 :     29      5      4      5      5      3      1      3      3      3      6   : 36668

 

05: 07  The rate was a bout 1500 and 2000, we contacted Oscar he said (if it's just bursts it should be ok), so they decided to do nothing.

 

06:26 

AIDA scalers attached 9

statistic attached 10

temretuer attached 11

bias attached 12

 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bf9 :      44  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0xa :      10  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     17     12      6      2      1      2      1      3      3      3      6   : 36228
aida02 :     19     11     12      3      2      2      2      4      2      3      6   : 36164
aida03 :     25      8     12      5      4      4      3      3      2      3      6   : 36356
aida04 :     22     19     11      3      1      2      3      4      2      3      6   : 36416
aida05 :     35      6      5      6      3      2      2      2      3      3      6   : 36236
aida06 :     12     11     11      2      3      4      1      3      3      3      6   : 36664
aida07 :     18      6      2      1      3      2      3      2      3      3      6   : 36216
aida08 :     27     10      8      3      0      1      2      3      3      3      6   : 36380
aida09 :     18      6      6      1      0      2      2      2      3      3      6   : 35832
aida10 :      3     14     10      3      1      1      2      3      3      3      6   : 36412
aida11 :      1      3      2      0      2      2      2      4      2      3      6   : 35772
aida12 :      0      5      6      3      3      4      1      3      3      3      6   : 36520

 

07:57  The beam stopped and they said there is a water leak !

 

 

 

 

 

Attachment 1: 20210418_0154.png
20210418_0154.png
Attachment 2: 20210418_0152_rate.png
20210418_0152_rate.png
Attachment 3: 20210418_0150_tem.png
20210418_0150_tem.png
Attachment 4: 20210418_0150_bias.png
20210418_0150_bias.png
Attachment 5: 20210418_0407.png
20210418_0407.png
Attachment 6: 20210418_0409.png
20210418_0409.png
Attachment 7: 20210418_0411.png
20210418_0411.png
Attachment 8: 20210418_0413.png
20210418_0413.png
Attachment 9: 18042021_0722_AIDA.png
18042021_0722_AIDA.png
Attachment 10: 18042021_0615_rate.png
18042021_0615_rate.png
Attachment 11: 18042021_0616_temp.png
18042021_0616_temp.png
Attachment 12: 18042021_0620_bias.png
18042021_0620_bias.png
  236   Sat Apr 17 19:39:56 2021 DJ, TDSaturday 17 April 20.00-00.00
19.37 per FEE64 rate spectra - attachments 1 & 2
      1.8.L spectra - attachments 3 & 4
      1.8.H spectra - attachments 5-8
      1.8.W spectra - attachments 9 & 10

20.42 DAQ contrinues file NULL/R30_233



		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bd7 : 	 10  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-------
23:48
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bde : 	 17  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x7 : 	 7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26      5      6      2      1      3      1      3      3      3      6   : 36336
aida02 :      7      9     10      3      2      2      2      4      2      3      6   : 36068
aida03 :     26      7      8      4      3      4      2      4      2      3      6   : 36448
aida04 :     18      8      2      1      2      2      3      4      2      3      6   : 36168
aida05 :     24      7      9      4      3      1      3      2      2      4      6   : 37352
aida06 :     17     10      3      1      3      5      1      3      3      3      6   : 36644
aida07 :      7     11      2      2      3      2      3      2      4      3      6   : 37268
aida08 :     23      4      1      5      1      1      2      3      3      3      6   : 36332
aida09 :     14      6      3      2      1      3      2      3      3      3      6   : 36504
aida10 :      2      2      2      2      1      2      2      2      3      3      6   : 35768
aida11 :      2      2      1      2      2      2      2      4      2      3      6   : 35816
aida12 :     26      8      5      5      4      3      1      3      3      3      6   : 36632
----
Attachment 1: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Attachment 7: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Attachment 8: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Attachment 9: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Attachment 10: Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 50.png
50.png
Attachment 13: 51.png
51.png
  235   Sat Apr 17 15:29:27 2021 MSSaturday 17 April 16:00-20:00 2021
16:21


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021




18:29
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bd1 : 	 4  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last




	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      3      4      4      1      3      1      3      3      3      6   : 36332
aida02 :      4      3     13      2      2      3      2      4      2      3      6   : 36152
aida03 :     12      7      5      2      2      3      2      4      2      3      6   : 36088
aida04 :     17      6      9      5      1      1      2      3      3      3      6   : 36452
aida05 :     26      8      0      3      1      1      2      3      3      3      6   : 36296
aida06 :     22      6      9      4      3      4      2      3      3      3      6   : 36952
aida07 :     17      7      3      1      2      3      3      2      3      3      6   : 36300
aida08 :     24      5      4      4      1      1      2      3      3      3      6   : 36360
aida09 :     15     11      3      2      1      3      1      3      3      3      6   : 36292
aida10 :      2      3      5      2      1      2      2      2      3      3      6   : 35824
aida11 :      2      2      0      0      3      2      2      4      2      3      6   : 35800
aida12 :     18     15      7      5      4      3      1      3      3      3      6   : 36688














	
Attachment 1: 40.png
40.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 6: 41.png
41.png
  234   Sat Apr 17 11:27:38 2021 JS, TDSaturday 17th April 12:00-16:00
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bce : 	 1  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

13:55 CEST

Statistics : ok elog:234/4
Temp : ok  elog:234/5
Bias : ok  elog:234/6
ucesb : ok
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and 
Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns


14:37 CEST

Statistics : ok
Temp : ok
Bias : ok   (ch3 now over 6uA)
ucesb : ok
DB: No faults found

15:03 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:31 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:50

Statistics : ok elog:234/7
Temp : ok  elog:234/8
Bias : ok  elog:234/9
ucesb : ok elog:234/10
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Attachment 1: 30.png
30.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 4: Screenshot_2021-04-17_Statistics_aidas-gsi.png
Screenshot_2021-04-17_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 31.png
31.png
Attachment 7: Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 32.png
32.png
Attachment 10: 33.png
33.png
  233   Sat Apr 17 07:28:02 2021 LPG, TDSaturday 08:00 - 12:00
08:30 CEST

HV and leakage currents: elog:233/1
Detector rates: elog:233/2
Temperatures: elog:233/3

WR status:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



FPGA status:


             Base         Current         Difference
aida12 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last





09:20 CEST

Stats: ok!
DB: No faults found
ucesb: ok!


09:40 CEST

Beam will be stopped in order to increase intensity. Expected to be around 3-4 hours.
For now, we still get implants when Beam spill is on. Seems to be fluctuating as they play around.

10:00 CEST

Still getting beam, it is fluctuating in intensity

Stats: ok!
DB: No faults found
ucesb: ok!

HV and leakage currents: elog:233/4
Detector rates: elog:233/5
Temperatures: elog:233/6

Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check: ok!
FPGA check:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory check: ok!


10:40 CEST

Stats: ok!
DB: No faults found
ucesb: ok!

11:00 CEST

Stats: ok!
DB: No faults found
ucesb: ok!


11:30 CEST

Stats: ok!
DB: No faults found
ucesb: ok!

12:00 BST

Stats: ok!
DB: No faults found
ucesb: ok!

HV and leakage currents: elog:233/7
Detector rates: elog:233/8
Temperatures: elog:233/9


Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
Memory check: ok!
Attachment 1: 20.png
20.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi.png
Screenshot_2021-04-17_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: 21.png
21.png
Attachment 5: Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 7: 22.png
22.png
Attachment 8: Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
  232   Fri Apr 16 23:13:57 2021 CA, MA, BASaturday 16th 00:00 - 08:00
00:05 DESPEC tuning beam, changing spill length

00:21 DAQ continues ok
      Merger/TapeServer ok

00:25 still no beam in AIDA

00:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 1

00:52 beam in AIDA - rates spectra - attachment 2

01:05 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 3

      beam is off again

01:30 beam back

01:32 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 4

01:37 beam off - FRS increasing spill length to 5s

01:48 beam back with new spill setting - 5s on spill, 2s off spill

02:07 system wide checks ok except;

WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb42 : 	 8  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check :

			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:10 FEE64 Temps ok - attachment 5

02:20 good event statistics ok - attachment 6

02:27 detector bias / leakage currents ok - attachment 7

02:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok

      beam off

03:00 stats/ucesb/database checks ok

03:30 stats/ucesb/database checks ok

      still no beam in AIDA

05:11 system is checked, beam is back with low intensity 

05:13 FEE64 Temps ok - attachment 8

05:13 good event statistics ok - attachment 9

05:13 detector bias / leakage currents ok - attachment 10

05:33 stats/ucesb/database checks ok

06:00 AIDA crashes, lose connection to aida09

06:20 aida08 does not respond on powercycle and reset

06:30 Called TD 

06:50 successfully reset AIDA. Writing to file R30

      synchronise ASIC clocks:
	
FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0

Database options file size check:


If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

System wide checks ok *except*

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

re-calibration of these FEE modules unsuccessful in FADC align and control.

07:05 FEE64 Temps ok - attachment 12

      good event statistics ok - attachment 13

      detector bias / leakage currents ok - attachment 14

07:10 stats/ucesb/database checks ok

07:30 stats/ucesb/database checks ok


      
      
Attachment 1: Screenshot_from_2021-04-16_23-35-20.png
Screenshot_from_2021-04-16_23-35-20.png
Attachment 2: Screenshot_from_2021-04-16_23-52-09.png
Screenshot_from_2021-04-16_23-52-09.png
Attachment 3: Screenshot_from_2021-04-17_00-05-06.png
Screenshot_from_2021-04-17_00-05-06.png
Attachment 4: Screenshot_from_2021-04-17_00-31-46.png
Screenshot_from_2021-04-17_00-31-46.png
Attachment 5: Screenshot_from_2021-04-17_01-11-18.png
Screenshot_from_2021-04-17_01-11-18.png
Attachment 6: Screenshot_from_2021-04-17_01-12-26.png
Screenshot_from_2021-04-17_01-12-26.png
Attachment 7: Screenshot_from_2021-04-17_01-13-22.png
Screenshot_from_2021-04-17_01-13-22.png
Attachment 8: 20210417_0506.png
20210417_0506.png
Attachment 9: 20210417_0602_tem.png
20210417_0602_tem.png
Attachment 10: 20210417_0601.png
20210417_0601.png
Attachment 11: 20210417_0603_Bias.png
20210417_0603_Bias.png
Attachment 12: Screenshot_from_2021-04-17_06-03-52.png
Screenshot_from_2021-04-17_06-03-52.png
Attachment 13: Screenshot_from_2021-04-17_06-04-14.png
Screenshot_from_2021-04-17_06-04-14.png
Attachment 14: Screenshot_from_2021-04-17_06-04-37.png
Screenshot_from_2021-04-17_06-04-37.png
  231   Fri Apr 16 15:52:08 2021 DSJ 16 April 16.00 shift
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :     25     10      4      3      1      2      2      3      3      3      6   : 36500
aida03 :     27      8      5      2      3      4      3      3      2      3      6   : 36092
aida04 :     30      4      4      5      0      2      4      2      3      3      6   : 36472
aida05 :     19      5      7      1      3      2      2      2      2      4      6   : 37060
aida06 :     18     13      1      2      1      3      2      3      3      3      6   : 36544
aida07 :     24      7      3      2      2      3      1      3      2      4      6   : 37384
aida08 :     15     11      2      5      2      4      1      4      3      3      6   : 37076
aida09 :      7      3      4      1      5      6      5      4      3      2      6   : 36308
aida10 :      8      3     13      4      3      3      1      4      2      3      6   : 36040
aida11 :     18     11     17      7      4      4      4      3      2      3      6   : 36752
aida12 :     12      7      7      7      4      4      2      3      2      3      6   : 36024



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021



17:27.  Checked system stats, temps, leakage etc. looks ok 


18:49. Tape server writing to disk /NULL/R21_9+
       /TapeData points to /media/SecondDrive/ - 3.6Tb free

19.15. checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9     16      2      2      2      1      3      3      3      6   : 36424
aida02 :      1      3      5      4      1      3      3      2      3      3      6   : 36268
aida03 :     24     13     11      6      3      3      2      4      2      3      6   : 36472
aida04 :      5      4      4      4      3      3      3      2      3      3      6   : 36404
aida05 :     17      8      4      2      2      2      2      2      2      4      6   : 36996
aida06 :     26     11      3      2      0      3      2      3      3      3      6   : 36528
aida07 :     24      7      2      1      1      1      2      3      2      4      6   : 37272
aida08 :     13     12      6      3      3      4      1      4      3      3      6   : 37140
aida09 :     22     15     19      2      6      5      4      4      3      2      6   : 36416
aida10 :     16     11      6      4      3      3      1      4      2      3      6   : 36024
aida11 :     12      4      5      4      2      3      3      3      2      3      6   : 35872
aida12 :     18     12      9      4      4      4      2      3      2      3      6   : 36024


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


19.53.  Checked system stats, temps, leakage etc. looks ok 

20:30.  Checked system stats, temps, leakage etc. looks ok 

20:59.  Checked system stats, temps, leakage etc. looks ok 



21:31 System checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      9     17      3      2      2      1      3      3      3      6   : 36492
aida02 :      5      1      1      6      0      2      2      3      3      3      6   : 36332
aida03 :      2     10     11      6      4      4      3      3      2      3      6   : 36296
aida04 :     27     12      3      4      0      2      3      2      3      3      6   : 36220
aida05 :     19      9      7      2      1      2      2      2      2      4      6   : 36996
aida06 :     28      5      0      4      0      3      1      3      3      3      6   : 36248
aida07 :     19     13     10      6      2      2      1      3      2      4      6   : 37524
aida08 :     18      9      6      0      1      4      1      3      3      3      6   : 36400
aida09 :     17     19     12      3      6      5      4      4      3      2      6   : 36348
aida10 :     23      9      7      4      2      3      1      4      2      3      6   : 35988
aida11 :     19     10      3      3      2      3      3      3      2      3      6   : 35884
aida12 :     12      6      0      3      2      4      2      3      2      3      6   : 35648


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


22.06.  Checked system stats, temps, leakage etc. looks ok 

22.18CEST BNC PB-5 amplitude changed from 1V to 2V
          fioler NULL/R21_67

22.23CEST All histograms zero'd

22.43.  Checked system stats, temps, leakage etc. looks ok 
23.21.  Checked system stats, temps, leakage etc. looks ok 


23.46 system checks
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     17     12      7      0      3      3      2      4      2      3      6   : 36180
aida02 :     35      3      5      2      0      2      1      3      3      3      6   : 36148
aida03 :     22      6     10      2      1      3      2      4      2      3      6   : 36136
aida04 :     27      9      3      5      2      4      3      3      2      3      6   : 36100
aida05 :     20      9      3      3      2      2      2      2      2      4      6   : 37032
aida06 :     25     11      1      2      0      3      1      3      3      3      6   : 36236
aida07 :     20      6      6      2      3      3      2      4      1      4      6   : 37216
aida08 :     25     10      0      2      1      3      1      3      3      3      6   : 36276
aida09 :     10      4      3      4      4      5      4      4      3      2      6   : 35960
aida10 :     26      9      3      0      1      3      1      4      2      3      6   : 35744
aida11 :      2      3      4      2      2      3      3      3      2      3      6   : 35744
aida12 :      4      9      7      2      2      4      2      3      2      3      6   : 35720
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021
Attachment 1: Screenshot_2021-04-16_Statistics_aidas-gsi.png
Screenshot_2021-04-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 10.png
10.png
Attachment 4: Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Attachment 5: 11.png
11.png
Attachment 6: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Attachment 8: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 9: 12.png
12.png
Attachment 10: Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 13.png
13.png
  230   Fri Apr 16 11:42:45 2021 OHShifter checklist
Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 2)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- Options database monitor (Top middle terminal screen 2, workspace 2)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 4)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 2)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 2)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 2)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 2)

- Perform the five system wide checks (Web browser tab screen 1, workspace 2)
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 4)
  229   Fri Apr 16 07:19:00 2021 OH, MAFriday 16th April 08:00-16:00
08:19 System wide checks all ok
      Baselines reset on WR and FPGA checks for the start of the day

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3

09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
      No corruptions have been noted since

10:10 System check, clcock and ADC ok

	 Base 		Current 	Difference
aida07 fault 	 0xfb3a : 	 0xfb3d : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA and memory check ok

      Statistics ok - attachment 4
      Temperatures ok - attachment 5
      Bias and leakage currents ok - attachment 6

10:39 Fission fragments were punching through AIDA
      This was even with the maxmimum degraders in place
      They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.

11:38 They have found the issue with the degrader and have corrected it.
      We now see next to no implants in AIDA as expect.
      Histograms reset at around 11:35
      Layout 2 shows very few implants in 5 minutes - attachment 7

12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
      This was a large amount of dose going into the detectors which we should try to avoid

13:25 system check, clock and ADC ok

	 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics ok - attachment 9
      Temperatures ok - attachment 10
      Bias and leakage currents ok - attachment 11
FPGA and memory check ok


16:10 system check, clock and ADC ok

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


      Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14
FPGA and memory check ok
Attachment 1: 210416_0820_Stats.png
210416_0820_Stats.png
Attachment 2: 210416_0821_Temp.png
210416_0821_Temp.png
Attachment 3: 210416_0824_Bias.png
210416_0824_Bias.png
Attachment 4: 210416_1003Stats.png
210416_1003Stats.png
Attachment 5: 210416_1007Temp.png
210416_1007Temp.png
Attachment 6: 210416_1019_voltages.png
210416_1019_voltages.png
Attachment 7: 210416_1139_layout2.png
210416_1139_layout2.png
Attachment 8: 210416_1207_Leakage.png
210416_1207_Leakage.png
Attachment 9: 210416_1311_Stats.png
210416_1311_Stats.png
Attachment 10: 210416_1312_Temp.png
210416_1312_Temp.png
Attachment 11: 210416_1315_Voltages.png
210416_1315_Voltages.png
Attachment 12: 210416_1412_Voltages.png
210416_1412_Voltages.png
Attachment 13: 210416_1548_Stats.png
210416_1548_Stats.png
Attachment 14: 210416_1553_Temp.png
210416_1553_Temp.png
  228   Fri Apr 16 05:13:09 2021 MLsystem wide checks

system wide checks:

Done around 4am.

#################################

To be noted:

-1- White rabbit and FPGA checks failed for 1 FEE and passed for 11

-2- No waveform for AIDA09 in 1.8.L spectrum

-3- AIDA 7 is red in the NewMerger control window (?)

#################################

 

Details:

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

White Rabbit checks:

         Base         Current     Difference
aida07 fault      0xfa8c :      0xfb37 :      171  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp check:


             Base         Current         Difference
aida07 fault      0x0 :      0xe :      14  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Memory check:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     45      9      7      3      4      3      2      3      3      3      6   : 36940
aida02 :     23      9      6      7      4      3      2      3      3      3      6   : 36964
aida03 :     40      8      5      4      5      3      2      3      3      3      6   : 36976
aida04 :     37     13      7      3      4      3      4      2      3      3      6   : 36940
aida05 :     35     11      6      3      4      4      2      3      2      4      6   : 38052
aida06 :     42      8      2      3      4      4      2      3      3      3      6   : 36968
aida07 :     42     10      5      5      5      3      2      3      3      3      6   : 37032
aida08 :     40      8      1      6      3      4      2      3      3      3      6   : 36976
aida09 :     18     18     14      8      5      5      5      4      3      2      6   : 36728
aida10 :      1      8      9      6      3      3      1      4      2      3      6   : 36052
aida11 :     24     11     14      6      2      2      4      2      3      3      6   : 36824
aida12 :     17     13      8      5      5      4      2      3      2      3      6   : 36108

Attachment 1: 16-04-2021_3h45am_HV.png
16-04-2021_3h45am_HV.png
Attachment 2: 48_temperature.png
48_temperature.png
Attachment 3: 16-04-2021_3h51_StatGoodEvt.png
16-04-2021_3h51_StatGoodEvt.png
Attachment 4: 16-04-2021_3h53_StatADCdataItem.png
16-04-2021_3h53_StatADCdataItem.png
Attachment 5: 16-04-2021_3h58_StatWR#4.png
16-04-2021_3h58_StatWR#4.png
Attachment 6: 16-04-2021_4h04_StatWR#5.png
16-04-2021_4h04_StatWR#5.png
Attachment 7: 16-04-2021_3h59_StatPause#2.png
16-04-2021_3h59_StatPause#2.png
Attachment 8: 16-04-2021_4h01_StatResume#3.png
16-04-2021_4h01_StatResume#3.png
Attachment 9: 16-04-2021_4h02_StatCorrelation#8.png
16-04-2021_4h02_StatCorrelation#8.png
Attachment 10: 16-04-2021_4h06_SpecRate.png
16-04-2021_4h06_SpecRate.png
Attachment 11: 16-04-2021_4h08_SpecRate2.png
16-04-2021_4h08_SpecRate2.png
Attachment 12: 16-04-2021_4h17_Spec1.8L.png
16-04-2021_4h17_Spec1.8L.png
Attachment 13: 16-04-2021_4h29_Spec1.8L_id4.png
16-04-2021_4h29_Spec1.8L_id4.png
Attachment 14: 16-04-2021_4h20_Spec1.8W_id7.png
16-04-2021_4h20_Spec1.8W_id7.png
Attachment 15: 25_Spec1.8W_id8.png
25_Spec1.8W_id8.png
Attachment 16: 16-04-2021_4h48_NewMergerControl.png
16-04-2021_4h48_NewMergerControl.png
  227   Fri Apr 16 02:15:12 2021 NHImplants/Punchthrough
Rate with beam < 1000 Hz still

Not Uranium but some fragments possibly produced in the thick 9g degrader
S4 slits are +/- 20mm but these punchthrough and cover the +/- 40mm of AIDA
Attachment 1: 53_AM.png
53_AM.png
  225   Fri Apr 16 00:47:36 2021 OH-MLAIDA DAQ Reset

1:18 (CET) No rate in AIDA 07.

Oscar did a AIDA DAQ reset and a quick wide check.

(Power cycle

All back to normal.

  224   Thu Apr 15 22:38:18 2021 OHThursday 15th April - 16th April
23:38 They are testing the finger detector
      It is a lot of U they are putting into it which is passing through to AIDA ~3500Hz
      Most seems to be punching through

23:45 Rate seems to be reduced to peaking at 1000Hz

00:06 DAQ Running to no storage
      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa


      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse
  223   Thu Apr 15 13:28:07 2021 TDThursday 15 April
13.28 System wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida01 fault 	 0x44c2 : 	 0x44c5 : 	 3  
aida02 fault 	 0xc81f : 	 0xc822 : 	 3  
aida03 fault 	 0x9342 : 	 0x9344 : 	 2  
aida04 fault 	 0xdb3f : 	 0xdb41 : 	 2  
aida05 fault 	 0x4baf : 	 0x4bb3 : 	 4  
aida06 fault 	 0xd2a4 : 	 0xd2a8 : 	 4  
aida07 fault 	 0xc51c : 	 0xc51f : 	 3  
aida08 fault 	 0xe853 : 	 0xe856 : 	 3  
aida09 fault 	 0x5212 : 	 0x5216 : 	 4  
aida10 fault 	 0xf81d : 	 0xf81f : 	 2  
aida11 fault 	 0x810 : 	 0x812 : 	 2  
aida12 fault 	 0xa279 : 	 0xa27a : 	 1  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x5 : 	 5  
aida12 fault 	 0x0 : 	 0xc : 	 12  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     57     33     12     15      3      3      3      2      3      2      6   : 35276
aida02 :     15      3      5      2      1      4      1      3      3      3      6   : 36388
aida03 :     13      4      2      1      4      3      1      3      3      3      6   : 36372
aida04 :     21      5      2      1      2      4      4      2      3      3      6   : 36668
aida05 :     10      5      6      3      2      3      3      3      1      4      6   : 36880
aida06 :     22     12      1      0      3      3      1      3      3      3      6   : 36360
aida07 :     19      3      3      0      3      3      2      4      1      4      6   : 37076
aida08 :     21      6      2      4      2      3      1      3      3      3      6   : 36388
aida09 :      2      7      5      3      2      4      2      3      2      3      6   : 35696
aida10 :      7      3      1      3      1      3      3      3      2      3      6   : 35684
aida11 :      1      2      1      3      2      5      2      2      3      3      6   : 36228
aida12 :     17     10      4      2      3      2      2      4      2      3      6   : 36052



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida02 =>   Options file size is 2114 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

13.30 Output of ~/oh/OptionsCheck.py

02:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&ASIC.se
ttings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

13.37 Detector bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachments 2
      Statistics - good events, ADC data, disc data, info code 4 & 5, pause & resume, correlation - attachments 3-10
      per FEE64 rate spectra - attachments 11 & 12
      per FEE64 1.8.L spectra - attachments 13 & 19
       aida5 peak width 70 ch FWHM, aida06 peak width 89ch FWHM
      per FEE64 1.8.W spectra 20us FSR - attachments 14 & 15
      meger & tape server - attachments 16-18
 
Attachment 1: 4.png
4.png
Attachment 2: Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-15_Statistics_aidas-gsi.png
Screenshot_2021-04-15_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Attachment 5: Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Attachment 6: Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Attachment 7: Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Attachment 8: Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Attachment 10: Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Attachment 11: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Attachment 14: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Attachment 15: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Attachment 16: Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Attachment 17: Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Attachment 18: 5.png
5.png
Attachment 19: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
  222   Thu Apr 15 12:23:22 2021 NHFast Discriminator Masks
It is possible to get good rates with the fast discriminators with a few hot channels mask in aida03 and aida10
Threshold is 0x20 = 3.2 MeV

They can be tested in beam to see performance, especially note the first setting will involve alpha emitters to check in AIDA

All FEEs:
AS01 01, AS3 07 10 13 are masked

aida03: 
AS0 12 and 14 are masked in addition

aida10:
AS1 10, 11 and AS2 05, 06 are masked in addiction
Attachment 1: Image_Pasted_at_2021-4-14_18-57.png
Image_Pasted_at_2021-4-14_18-57.png
  221   Wed Apr 14 15:14:55 2021 PJCSNote Well

Please note that for the forseeable future the system function in the menu on the System Wide Checks browser page labelled as 

"Collect the Timestamp RAM values"  and "Start the Readout Timestamp error tracing" 

Are for the use of engineers only unless otherwise instructed. 

There is no harm in using them but the results may be confusing and it is not possible, at present, to detail a definitive set of instructions for their use.

  220   Wed Apr 14 12:14:47 2021 PJCSCorrections and changes

14/4/21 @12:00 UK time

Corrected the failure of FADC re-calibration All modules. The problem was a missing > in the .tml file

Edited the sys.tcl file in /MIDAS/TclHttpd/Html/RunControl to comment out where the status of the waveform enable was able to stop the creation of the .W spectra. This will now function at SETUP.

Edited the sys.tcl file to remove the test for Master module at the enable of the Correlation Scalar readout. All modules will now be enabled.

Successfully re-calibrated the LMK3200 in aida09 to remove the clock fault reported in System Wide Checks.

Successfully "sync ASIC clocks" followed by calibrate all module FADCs ( except aida09 which required seperate attention. )

All the Edits first carried out and tested at Daresbury T9 system.

  219   Wed Apr 14 09:55:12 2021 TDTuesday 13 April
Two further comments

1) The 'pristine' Options files produced by OH fixed *all* of the undefined spectra issues,
   i.e. we now have *L, *H, Rate, Stat, Hit, HitRate spectra for all FEE64s

2) The system has undegone 1x power cycle/reset/setup and 1x reset/setup cycle since the
   new Options files were installed and we currently have 2x errors reported in aida02.
   There is an open Options tab and we have viewed the Options settings for aida01 and aida02
   (currently viewing aida02) - *no* changes/saves/restores requested.

> 
> Rename Options directory
> 
> mv Options Options.BAK-130421
> 
> Copy Options.PRISTINE to Options
> 
> cp -r Options.PRISTINE Options
> chmod -R 777 Options
> 
> Start Options directory monitor (will check directory every 5 minutes)
> 
> Python ~/oh/OptionsCheck
> 
> 
> 17.32 Power cycle FEE64s
> 
> 
> FEE64 module aida09 global clocks failed, 6
>  Clock status test result: Passed 11, Failed 1
> 
> Understand status as follows
> Status bit 3 : firmware PLL that creates clocks from external clock not locked
> Status bit 2 : always logic '1' 
> Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
> Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
> If all these bits are not set then the operation of the firmware is unreliable
> 
> Calibration test result: Passed 12, Failed 0
> 
> If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
> 
> White Rabbit error counter test result: Passed 12, Failed 0
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 	
> FPGA Timestamp error counter test result: Passed 12, Failed 0
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
> 
> 
> Returned 0 0 0 0 0 0 0 0 0 0 0 0  
> Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
> aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
> aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
> aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
> aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
> aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
> aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
> aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
> aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
> aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
> aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116
> 
> 
> Note that system wide checks identify 1x file of different size immediately after boot/reset/setup
> 
>  Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
> 
>  FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021
> 
> 06:25:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> 06:30:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.setting
s&&WAVE_DMA_H
> 
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_En
able
> 
> 
> 
> 
> 
> FEE64 module aida01  => 	7
> FEE64 module aida02  => 	7
> FEE64 module aida03  => 	7
> FEE64 module aida04  => 	7
> FEE64 module aida05  => 	7
> FEE64 module aida06  => 	7
> FEE64 module aida07  => 	7
> FEE64 module aida08  => 	7
> FEE64 module aida09  => 	7
> FEE64 module aida10  => 	7
> FEE64 module aida11  => 	7
> FEE64 module aida12  => 	7
> 
> 
> ASIC Clock Timestamp check test result: Passed 12, Failed 0
> 
> 
>  If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status
> 
> 
> 
> 
>  aida01 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida02 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida03 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida04 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida05 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida06 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida07 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida08 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida09 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida10 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida11 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida12 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
> 17.14 rate spectra - attachment 1
> 
>       1.8.L spectra - attachments 2 & 3
>       pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM
> 
>       1.8.W spectra - attachments 4 & 5
> 
>       good events statistics (slow comparator 0xa) - attachment 6
> 
>       FEE64 temperatures OK - attachment 7
>     
>       DSSSD bias & leakage currents OK - attachment 8
> 
> 
> 
> 
> 17.38 DAQ starts file NULL/R14
>       slow comparator 0xa -> 0x64
>       BNC PB-5 pulser OFF
>       alpha background
> 
> 10:40 CET 14.04.2021 - NH
>       Close file NULL/R14
  218   Tue Apr 13 16:49:41 2021 TDTuesday 13 April
Rename Options directory

mv Options Options.BAK-130421

Copy Options.PRISTINE to Options

cp -r Options.PRISTINE Options
chmod -R 777 Options

Start Options directory monitor (will check directory every 5 minutes)

Python ~/oh/OptionsCheck


17.32 Power cycle FEE64s


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116


Note that system wide checks identify 1x file of different size immediately after boot/reset/setup

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021

06:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

06:30:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable





FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0


 If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status




 aida01 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida02 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida03 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida04 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida05 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida06 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida07 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida08 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida09 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida10 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida11 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida12 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

17.14 rate spectra - attachment 1

      1.8.L spectra - attachments 2 & 3
      pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM

      1.8.W spectra - attachments 4 & 5

      good events statistics (slow comparator 0xa) - attachment 6

      FEE64 temperatures OK - attachment 7
    
      DSSSD bias & leakage currents OK - attachment 8




17.38 DAQ starts file NULL/R14
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      alpha background

10:40 CET 14.04.2021 - NH
      Close file NULL/R14
Attachment 1: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-13_Statistics_aidas-gsi.png
Screenshot_2021-04-13_Statistics_aidas-gsi.png
Attachment 7: Screenshot_2021-04-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 8: 3.png
3.png
  217   Tue Apr 13 14:24:48 2021 TDS460 information
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S460 shift plan - https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1097596683

Mattermost - https://mattermost.gsi.de/despec/channels/aida

DESPEC Zoom meeting

Zoom Meeting https://gsi-fair.zoom.us/j/94404952361
Passcode: s460
  216   Mon Apr 12 18:36:31 2021 NH[HowTo] AIDA & MBS
Some instructions on AIDA and MBS and troubleshooting are available here
https://sf.gsi.de/lib/c9ba0b4c-26bc-43d0-9b49-213fc594610f/file/AIDA%20and%20MBS%20Tips%20and%20Troubleshooting.pdf
Attachment 1: AIDA_and_MBS_Tips_and_Troubleshooting.pdf
AIDA_and_MBS_Tips_and_Troubleshooting.pdf AIDA_and_MBS_Tips_and_Troubleshooting.pdf AIDA_and_MBS_Tips_and_Troubleshooting.pdf
  215   Mon Apr 12 16:49:40 2021 TDMonday 22 April - pulser walkthrough & alpha background
16.54 Pulser walkthrough

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      file NULL/R4

      BNC PB-5 settings changed by terminal command line using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      aida07 & aida08 1.8.L spectra - see attachment 1

17.26 run ends OK file NULL/R4_27

17.26 file NULL/R5

      amplitudes 0.6, 1, 1.4, 1.8, 2.2, 2.6, 3

      aida07 & aida08 1.8.L spectra - see attachment 2

17.47 run ends OK NULL/R5_23

17.53 system wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     11      3      2      0      2      4      2      3      4     11   : 58888
aida02 :      7     11      4      2      0      4      3      2      3      3      7   : 40436
aida03 :     38      7      7      2      6      4      1      3      3      3      6   : 36864
aida04 :     19     10      5      2      1      3      1      4      2      4     15   : 74732
aida05 :     30      5      4      3      2      3      4      3      3      4     15   : 76096
aida06 :     24      4      3      3      1      4      2      3      2      4      7   : 41808
aida07 :     22      7     11      7      1      6      3      4      3      3      6   : 37984
aida08 :     13      7      5      2      1      4      3      3      1      4      6   : 36924
aida09 :     33      9     10      6      4      4      2      2      2      4      6   : 37676
aida10 :     15      7      7      1      1      3      1      3      2      3      7   : 39364
aida11 :     30     13      8      3      4      4      3      3      3      3      6   : 37312
aida12 :     17      9      4      3      2      3      2      2      3      3      7   : 40236

17.57 Slow comparator 0xa -> 0x64
      BNC PB-5 Pulser OFF

      file NULL/R9
      alpha background

      FEE64 temperatures OK - see attachment 3
      statistics OK - see attachment 4
      DSSSD bias & leakage currents OK - see attachment 5
      tape server & merger - attachment 6

11:31 13.04.21 
      file NULL/R9 stopped for work in day
      will resume alpha over night again

 
Attachment 1: Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-12_Statistics_aidas-gsi.png
Screenshot_2021-04-12_Statistics_aidas-gsi.png
Attachment 5: 1.png
1.png
Attachment 6: 2.png
2.png
  214   Sat Apr 10 12:15:54 2021 OH, TDSaturday 10 April
12:37 DAQ found crashed. Had crashed at some time following 7:50am following previous statistics update
      Unable to recover the FEEs so forced to power cycle
      Following power cycle statistics much improved from yesterday.
      In the waveforms the lower frequency component that was prevalent yesterday is no longer visible.
      The waveforms are not perfect by any means though
      Also note that following an ASIC synchronisation all of the FEEs lose ADC calibration and must be manually recalibrated.

16.42 UTC+1

      per FEE64 1.8.W spectra

      20us FSR attachments 7 & 8
      200us FSR attachments 9 & 10
      2ms FSR attachments 11 & 12
      20ms FSR attachments 15 & 16


      rate spectra attachment 13
      good events statistics attachment 14
Attachment 1: 210410_1237_Aida_crash.png
210410_1237_Aida_crash.png
Attachment 2: 210410_1242_Crash.png
210410_1242_Crash.png
Attachment 3: 210410_1251_aida06_drop.png
210410_1251_aida06_drop.png
Attachment 4: 210410_1312_Waveform1.png
210410_1312_Waveform1.png
Attachment 5: 210304_1313_Waveform2.png
210304_1313_Waveform2.png
Attachment 6: 210410_1318_Stats.png
210410_1318_Stats.png
Attachment 7: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 10: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 12: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 14: Screenshot_2021-04-10_Statistics_aidas-gsi.png
Screenshot_2021-04-10_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 16: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
  213   Fri Apr 9 16:45:29 2021 NH, HA. GA, GZ, OH, TDAIDA Noise
AIDA snout was reconnected and we saw originally a very good reduction in noise from S452 (fig 1)
Occasional bursts of increased rates were noticed, perhaps related to works in S4.

Tape rate at optimal conditions was around 6 MB/s
We still saw some extrinsic noise in waves, two frequencies observed : 1.5 MHz and 100 kHz or so

Figs 2, 3 are at 50 MS/s
Figs 4, 5 are at 5 MS/s

Today the situation has become a lot worse for reasons still unknown.
aida01 shows very strong noise, frequency 125 kHz or so it seems (still at 5 MS/s)
Amplitude 100 mV or so I think

Investigation still underway
Attachment 1: Image_Pasted_at_2021-4-7_18-15.png.jpeg
Image_Pasted_at_2021-4-7_18-15.png.jpeg
Attachment 2: Image_Pasted_at_2021-4-8_16-142.png.jpeg
Image_Pasted_at_2021-4-8_16-142.png.jpeg
Attachment 3: Image_Pasted_at_2021-4-8_16-14.png.jpeg
Image_Pasted_at_2021-4-8_16-14.png.jpeg
Attachment 4: Image_Pasted_at_2021-4-8_16-17.png.jpeg
Image_Pasted_at_2021-4-8_16-17.png.jpeg
Attachment 5: Image_Pasted_at_2021-4-8_16-18.png.jpeg
Image_Pasted_at_2021-4-8_16-18.png.jpeg
Attachment 6: noise_aida_090421.png
noise_aida_090421.png
  212   Fri Apr 9 16:40:36 2021 NHnnrpi1 Update
nnrpi1 is now back in S4 with a new SD card and new raspberry pi.
The freezing originally continued (it was the USB system failing) but was fixed with a change to /boot/cmdline.txt coherent_pool=4M
It seems the kernel update to fix one bug introduced another... 
Now it is running again with all TTYs connected. I think one TTY may not be working but 11 should be
  211   Wed Mar 31 15:46:09 2021 PJCSHowTo : Synchronize the ASIC clocks.

To Synchronize the FEE64 ASIC clocks to rise at the same timestamp time use the System function Synchronise the ASIC clocks in the System Wide Checks browser page.

The Server will read the current timestamp value and calculate, based on the number of FEEs and the access delay, a timestamp value sometime in the future. F_stamp

F_stamp is written to each FEE and the synchronization is enabled. 

In a state machine in the FEE ASIC clock control, logic is enabled to synchronize its ASIC clock. It compares the current timestamp with F_stamp and if the current timestamp is >= F_stamp then it will start the ASIC clock.

The report in the browser log window gives the 3 LSBs of the timestamp at the instant the ASIC clock is started. They should all be the same. They are from the 10ns timestamp.

 

  210   Wed Mar 31 15:40:42 2021 PJCSCheck Options files function added

There is a new operation available in the System Functions menu in the System Wide Checks page. ( The page needs to be reset once when the FEEs are powered before this function will appear )

Check the Options files are all the same size

Using this function will provide a list of the file sizes of the Options/< fee name >/CONTENTS files along with the last time they were accessed as a list.

It's up to the user, at present, to interpret the results but more is possible if required ....

  209   Wed Mar 31 12:46:10 2021 PJCSHowTo : Calibrate the LMK3200 clock devices

Here is a document showing how and where to calibrate the LMK3200 devices that lock to the system clock and generate the clocks for the ADCs and the FPGA internal logic.

Attachment 1: Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
  208   Tue Mar 30 13:21:50 2021 NHnnrpi1 (tty logs)
We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one.
Trying to reboot the pi to detect the new USB ports and it didn't recover from boot
Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead
A new SD card will be ordered and nnrpi1 restored from the original images

A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it)
  207   Thu Mar 25 18:04:51 2021 TDMSL type BB18(DS) 24cm x 8cm cabling & bias configuration
Attachment 1: MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf
MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf
  206   Wed Mar 24 10:31:03 2021 TDAIDA to do (or to understand) list
In no particular order

1) Options

variables unexpectedly changing or becoming unset/undefined - see https://elog.ph.ed.ac.uk/DESPEC/205

Is this why *.L, *.H and Rate spectra are undefined in some FEE64s?


2) Merger 'Bad Timestamp' error reports - e.g. see https://elog.ph.ed.ac.uk/DESPEC/203

Meaning of report? Significance/impact?


3) Large numbers of PAUSE/RESUME for some (but not all) FEE64s - see e.g. https://elog.ph.ed.ac.uk/DESPEC/203

Significance/impact?

-- NH to add

4) Slow but non-zero increase of White Rabbit errors (mostly timestamp but much rarer ASIC errors)
We know the White Rabbit network in general is reliable but could we see interruption to the 200 MHz clock or 10 kHz timestamp when FATIMA DAQ is busy/restarting
Wasn't uniform over all FEEs implying a more local source though?

-- OH to add
5) The dropping of links between FEEs and the MERGER. Quite often these were correlated with the 'Bad Timestamp' error messages and also the WR errors

---- PJCS to add
6) The addition of the NIM clock-divider to allow better ASIC clock synchronisation.
  205   Fri Mar 19 17:16:45 2021 TDFriday 19 March
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.
Attachment 1: Screenshot_2021-03-19_Options_aidas-gsi.png
Screenshot_2021-03-19_Options_aidas-gsi.png
Attachment 2: Screenshot_2021-03-19_Options_aidas-gsi(4).png
Screenshot_2021-03-19_Options_aidas-gsi(4).png
Attachment 3: Screenshot_2021-03-19_Options_aidas-gsi(1).png
Screenshot_2021-03-19_Options_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-19_Options_aidas-gsi(5).png
Screenshot_2021-03-19_Options_aidas-gsi(5).png
Attachment 5: Screenshot_2021-03-19_Options_aidas-gsi(2).png
Screenshot_2021-03-19_Options_aidas-gsi(2).png
Attachment 6: Screenshot_2021-03-19_Options_aidas-gsi(6).png
Screenshot_2021-03-19_Options_aidas-gsi(6).png
Attachment 7: Screenshot_2021-03-19_Options_aidas-gsi(3).png
Screenshot_2021-03-19_Options_aidas-gsi(3).png
Attachment 8: Screenshot_2021-03-19_Options_aidas-gsi(7).png
Screenshot_2021-03-19_Options_aidas-gsi(7).png
  204   Thu Mar 18 11:53:10 2021 TDThursday 18 March
12.40 The snout w/bPlast plus 3xDSSSDs was removed yesterday

      AIDA pulser tests via adaptor PCB test inputs, i.e. zero capacitative load and zero leakage current

      per FEE64 1.8.W spectra - attachments 1-4
      per FEE64 Rate spectra - attachment 5
      per FEE64 1.8.L spectra - attachments 6-7 

From the per FEE64 1.8.L

FEE64 FWHM

1  -
2  45
3  25
4  -
5  -
6  45
7  25
8  37
9  30
10 38
11 19
12 35

For comparison we would expect to obtain ~ 15-20 ch FWHM (~10.5-14 keV FWHM) based on tests
conducted at T9, STFC DL. The peak widths observed above are significantly worse than expected.
Are the adaptor PCBs currently grounded to FEE64 Cu mezzanine cover?
Attachment 1: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(5).png
Attachment 7: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(6).png
  203   Wed Mar 17 10:31:09 2021 TDWednesday 17 March
11.28 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks
	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25     16      3      6      1      3      2      2      3      4     10   : 54676
aida02 :     17     13     14     10      5      5      2      3      3      3      6   : 37516
aida03 :     19     10      8      3      2      3      2      4      2      3      6   : 36220
aida04 :      2      6      2      3      3      2      2      3      2      4     15   : 74360
aida05 :     89     48     20      3      3     34     21     20      6      4      7   : 64324
aida06 :     31     11      1     10      0      1      1      3      4      2      5   : 31140
aida07 :     21     13      8      4      3      2      4      4      3      3      6   : 37756
aida08 :     37     10      2      0      1      2      2      4      3      3      6   : 36932
aida09 :      4     18      6      3      1      3      1      2      3      3      6   : 35872
aida10 :     20      4      1      3      0      2      2      3      3      3      6   : 36320
aida11 :      2      6      6      2      2      4      2      3      2      3      6   : 35672
aida12 :     15     10      2      3      1      2      2      3      3      3      6   : 36428


      FEE64 Temperatures OK - attachment 1
      Good event, disc info #6 & good wave statistics - attachments 2, 3 & 4
           note high disc rates (c. 200k) in FEE64s aida03, aida05 & aida07
      Detector bias & leakage currents OK - attachment 5
      Merger OK - 4.5M data items/s
      TapeServer OK - 15Mb/s

      Rate spectra - attachment 6

11.41 Merger errors since yesterday's Elog


MERGE Data Link (20502): bad timestamp  2 3 0xc08e4cce 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f4f 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0807dc7 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0957ee9 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0857eea 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0977df3 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc08a7edd 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7c7f 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7fd8 0x0c1cb340 0x0000f14c5c1cb340 0x166cf14c5c1cb340 0x166cf1515c1c74c0

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c338 0x000018ff4433c338 0x166d18ff4433c338 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c356 0x000018ff4433c356 0x166d18ff4433c356 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263f080 0x0433c36a 0x000018ff4433c36a 0x166d18ff4433c36a 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82630f7f 0x0433c374 0x000018ff4433c374 0x166d18ff4433c374 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c388 0x000018ff4433c388 0x166d18ff4433c388 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3a6 0x000018ff4433c3a6 0x166d18ff4433c3a6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c3c4 0x000018ff4433c3c4 0x166d18ff4433c3c4 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3d8 0x000018ff4433c3d8 0x166d18ff4433c3d8 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3f6 0x000018ff4433c3f6 0x166d18ff4433c3f6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c414 0x000018ff4433c414 0x166d18ff4433c414 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82218ff4 0x0433c41e 0x000018ff4433c41e 0x166d18ff4433c41e 0x166d18ff4e17c2c0

From Tuesday, March 16, 2021 9:54:36.584 PM to Wednesday, March 17, 2021 10:02:05.935 AM

11.46 Grafana record of bias & leakage current for the 7 days to date - attachment 7

11.51 ASIC check load - all FEE64s, all ASICs

11.51 Disable 'no storage' mode and start writing to disk file S452/R58

12.01 R58_18 & R58_19 *no* change

12.16  R58_24 & R58_25 *all* waveforms disabled (Run Control tab) 

12.23 R58_27 & R58_28 *all* discriminators disabled (Discriminator tab) *but* disc rates now low - see attachment 8 cf. attachment 3

12.27 Enable 'no storage' mode

12.38 Analysis of files R58_19, R58_25 & R58_28 - see attachments 9, 10 & 11
      Disabling waveforms significantly reduces pause/resume for aida01 - aida08
      Effect of disabling discriminators less clear - disc rate zero for all FEE64s except aida03 (2k).
Attachment 1: Screenshot_2021-03-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-17_Statistics_aidas-gsi.png
Screenshot_2021-03-17_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-03-17_Statistics_aidas-gsi(1).png
Screenshot_2021-03-17_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-17_Statistics_aidas-gsi(2).png
Screenshot_2021-03-17_Statistics_aidas-gsi(2).png
Attachment 5: 84.png
84.png
Attachment 6: Screenshot_2021-03-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-17_Spectrum_Browser_aidas-gsi.png
Attachment 7: 85.png
85.png
Attachment 8: Screenshot_2021-03-17_Statistics_aidas-gsi(3).png
Screenshot_2021-03-17_Statistics_aidas-gsi(3).png
Attachment 9: R58_19
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260603204 ( 1825295.6 Hz)
        Other data format:    1316796 (    9223.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1800 (      12.6 Hz)
                               RESUME:       1800 (      12.6 Hz)
                              SYNC100:        532 (       3.7 Hz)
                              WR48-63:        532 (       3.7 Hz)
                           FEE64 disc:     351517 (    2462.1 Hz)
                             MBS info:     960615 (    6728.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      142.773 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.872              137.171
  2                2.370              140.392
  3                4.557              140.929
  4               12.507              102.274
  5                9.566              138.244
  6                0.000              130.460
  7                0.043              132.607
  8               21.086              137.171
  9               14.355              142.539
 10                0.094              126.165
 11               38.783              127.507
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21184711       1567          0          0        302        302         78         78          0        807          0          0
  1   26163497        939          0          0          5          5         40         40          0        849          0          0
  2   26414580     834315          0          0          9          9         78         78     351513     482628          0          0
  3   30144417     476485          0          0         23         23         54         54          0     476331          0          0
  4    6641111       1448          0          0        716        716          8          8          0          0          0          0
  5   18794955        716          0          0        334        334         24         24          0          0          0          0
  6   13042480         36          0          0          0          0         18         18          0          0          0          0
  7    9502247         42          0          0          1          1         20         20          0          0          0          0
  8   31490514        302          0          0        109        109         42         42          0          0          0          0
  9   33799793        352          0          0         86         86         88         88          4          0          0          2
 10   11018662         96          0          0          2          2         46         46          0          0          0          0
 11   32406237        498          0          0        213        213         36         36          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45075.926s (    0.710 blocks/s,   0.044 Mb/s)
Attachment 10: R58_25
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261566975 ( 1848651.9 Hz)
        Other data format:     353025 (    2495.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        498 (       3.5 Hz)
                               RESUME:        497 (       3.5 Hz)
                              SYNC100:        528 (       3.7 Hz)
                              WR48-63:        528 (       3.7 Hz)
                           FEE64 disc:     350128 (    2474.6 Hz)
                             MBS info:        846 (       6.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      141.491 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.252              140.123
  2                1.261              137.707
  3                1.206              132.607
  4                0.000              122.675
  5                0.045              124.017
  6                0.225              128.312
  7                0.137              137.171
  8               25.852              127.238
  9               22.485              137.976
 10                0.119              139.318
 11               34.852              135.828
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22601243       1010          0          0          5          5         77         77          0        846          0          0
  1   25817640        118          0          0         10         10         49         49          0          0          0          0
  2   26662668     350294          0          0         10         10         73         73     350128          0          0          0
  3   31065481        116          0          0          9          9         49         49          0          0          0          0
  4    7606752         38          0          0          0          0         19         19          0          0          0          0
  5   20370392         52          0          0          3          3         23         23          0          0          0          0
  6   13822142         52          0          0          5          5         21         21          0          0          0          0
  7    9358078         60          0          0          5          5         25         25          0          0          0          0
  8   31862906        361          0          0        132        131         49         49          0          0          0          0
  9   30480452        364          0          0        115        115         67         67          0          0          0          0
 10    7772769         92          0          0          5          5         41         41          0          0          0          0
 11   34146452        468          0          0        199        199         35         35          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45119.996s (    0.709 blocks/s,   0.044 Mb/s)
Attachment 11: R58_28
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261917114 ( 1867522.0 Hz)
        Other data format:       2886 (      20.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        500 (       3.6 Hz)
                               RESUME:        500 (       3.6 Hz)
                              SYNC100:        523 (       3.7 Hz)
                              WR48-63:        523 (       3.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:        840 (       6.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          1 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      140.248 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.196              135.828
  2                1.735              133.949
  3                1.362              133.412
  4                0.000              133.949
  5                0.624              134.218
  6                0.297              110.595
  7                0.085              138.513
  8               26.310              131.265
  9               16.413              138.781
 10                0.037              139.318
 11               35.860              131.265
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22486922       1022          0          0          7          7         84         84          0        840          0          0
  1   25724429         84          0          0          9          9         33         33          0          0          0          0
  2   26397664        186          0          0          7          7         86         86          0          0          0          0
  3   31045386        104          0          0         11         11         41         41          0          0          0          0
  4    7561349         28          0          0          0          0         14         14          0          0          0          0
  5   20305012         70          0          0          8          8         27         27          0          0          0          0
  6   14228124         40          0          0          6          6         14         14          0          0          0          0
  7    9428091         66          0          0          3          3         30         30          0          0          0          0
  8   31422421        379          0          0        144        145         45         45          0          0          0          0
  9   32040015        335          0          0         92         91         76         76          0          0          0          1
 10    7197353         72          0          0          2          2         34         34          0          0          0          0
 11   34080348        500          0          0        211        211         39         39          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45164.730s (    0.709 blocks/s,   0.044 Mb/s)
  202   Tue Mar 16 09:26:47 2021 TDTuesday 16 March
10.25 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     19      5      5      1      3      2      2      3      4     10   : 54660
aida02 :     20     13     14     11      5      5      2      4      2      3      6   : 37048
aida03 :      2      3      7      3      2      3      2      4      2      3      6   : 36080
aida04 :     14     10      4      3      3      2      2      4      1      4     15   : 73960
aida05 :     97     51     19      3      3     34     21     20      6      4      7   : 64364
aida06 :     24      7      5      9      0      1      1      3      4      2      5   : 31112
aida07 :     21     13      9      3      3      2      3      4      3      3      6   : 37484
aida08 :     38     10      3      5      2      3      1      4      3      3      6   : 37048
aida09 :     24     11      4      3      2      3      2      3      3      3      6   : 36696
aida10 :     26     10      1      3      1      2      1      4      3      3      6   : 36712
aida11 :      0      7      7      1      1      4      2      3      2      3      6   : 35592
aida12 :      6      8      3      2      2      1      2      3      3      3      6   : 36296


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.5M data items/s
      TapeServer OK - 13Mb/s

      No merger server error messages since most recent check yesterday

15.50 Merger error messages


 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0x838153cb 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d481de 0x090bff32 0x0000c2c4f90bff32 0x166cc2c4f90bff32 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c181f8 0x090c0ed2 0x0000c2c4f90c0ed2 0x166cc2c4f90c0ed2 0x166cc2c4fefc5fd2
MERGE Actor (20516): Working with 0 from 12 data sources

MERGE Data Link (20503): bad timestamp  3 3 0xc0c47f9e 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d08127 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c681ee 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181c9 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98204 0x090c2642 0x0000c2c4f90c2642 0x166cc2c4f90c2642 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8075 0x090c2e12 0x0000c2c4f90c2e12 0x166cc2c4f90c2e12 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf812f 0x090c35e2 0x0000c2c4f90c35e2 0x166cc2c4f90c35e2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780ef 0x090c4d52 0x0000c2c4f90c4d52 0x166cc2c4f90c4d52 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f7816c 0x090c8bd2 0x0000c2c4f90c8bd2 0x166cc2c4f90c8bd2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d480f1 0x090c93a2 0x0000c2c4f90c93a2 0x166cc2c4f90c93a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c48039 0x090ca342 0x0000c2c4f90ca342 0x166cc2c4f90ca342 0x166cc2c4fefc5fd2

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0xc08d7cc1 0x085c7e10 0x0000c429685c7e10 0x166cc429685c7e10 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e281c 0x085c85e0 0x0000c429685c85e0 0x166cc429685c85e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7cc1 0x085c8db0 0x0000c429685c8db0 0x166cc429685c8db0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e8c 0x085c9580 0x0000c429685c9580 0x166cc429685c9580 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c414b 0x085ca520 0x0000c429685ca520 0x166cc429685ca520 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dee 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987d2c 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d90 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fd6 0x085d02e0 0x0000c429685d02e0 0x166cc429685d02e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e7c 0x085d2220 0x0000c429685d2220 0x166cc429685d2220 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857d7f 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987cc0 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ccd 0x085d5100 0x0000c429685d5100 0x166cc429685d5100 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e80fd 0x085d7fe0 0x0000c429685d7fe0 0x166cc429685d7fe0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e77 0x085d8f80 0x0000c429685d8f80 0x166cc429685d8f80 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0817ec8 0x085d9750 0x0000c429685d9750 0x166cc429685d9750 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0847fca 0x085d9f20 0x0000c429685d9f20 0x166cc429685d9f20 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c40e8 0x085da6f0 0x0000c429685da6f0 0x166cc429685da6f0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927f3e 0x085db690 0x0000c429685db690 0x166cc429685db690 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0x82614000 0x085dbb36 0x0000c429685dbb36 0x166cc429685dbb36 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857ebc 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987eec 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380

21.04 Merger errors since 15.50

MERGE Data Link (20502): bad timestamp  2 3 0xc0907de6 0x03087130 0x0000ea6805fd8d80 0x166cea6803087130 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0878007 0x030880d0 0x0000ea68030880d0 0x166cea68030880d0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937ec9 0x030888a0 0x0000ea68030888a0 0x166cea68030888a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4e2d 0x0308a7e0 0x0000ea680308a7e0 0x166cea680308a7e0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0998001 0x0308bf50 0x0000ea680308bf50 0x166cea680308bf50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4abc 0x0308fdd0 0x0000ea680308fdd0 0x166cea680308fdd0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0907fb1 0x030905a0 0x0000ea68030905a0 0x166cea68030905a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937fd8 0x03091d10 0x0000ea6803091d10 0x166cea6803091d10 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0807df1 0x03093480 0x0000ea6803093480 0x166cea6803093480 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d14 0x03093c50 0x0000ea6803093c50 0x166cea6803093c50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x8280dbb0 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82816a12 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e07 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0967c2a 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0887d0f 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dad 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7bfc 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c98 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7f7e 0x03096b30 0x0000ea6803096b30 0x166cea6803096b30 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e10 0x03098a70 0x0000ea6803098a70 0x166cea6803098a70 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0957f91 0x048638a0 0x0000ea6bf48638a0 0x166cea6bf48638a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967e3d 0x04864070 0x0000ea6bf4864070 0x166cea6bf4864070 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4dd3 0x04865fb0 0x0000ea6bf4865fb0 0x166cea6bf4865fb0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0998027 0x04866780 0x0000ea6bf4866780 0x166cea6bf4866780 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7dd9 0x0486a600 0x0000ea6bf486a600 0x166cea6bf486a600 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d45 0x0486add0 0x0000ea6bf486add0 0x166cea6bf486add0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7e98 0x0486b5a0 0x0000ea6bf486b5a0 0x166cea6bf486b5a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967d19 0x0486cd10 0x0000ea6bf486cd10 0x166cea6bf486cd10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e95 0x0486ec50 0x0000ea6bf486ec50 0x166cea6bf486ec50 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dba 0x0486f420 0x0000ea6bf486f420 0x166cea6bf486f420 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c2e 0x04874240 0x0000ea6bf4874240 0x166cea6bf4874240 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ea0 0x04874a10 0x0000ea6bf4874a10 0x166cea6bf4874a10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4b7e 0x048780c0 0x0000ea6bf48780c0 0x166cea6bf48780c0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dd8 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947ed6 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7f9f 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f8e 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d73 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d81 0x0487b770 0x0000ea6bf487b770 0x166cea6bf487b770 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7d98 0x0487bf40 0x0000ea6bf487bf40 0x166cea6bf487bf40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09d803b 0x0487c710 0x0000ea6bf487c710 0x166cea6bf487c710 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4ca6 0x0487d6b0 0x0000ea6bf487d6b0 0x166cea6bf487d6b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dba 0x04880d60 0x0000ea6bf4880d60 0x166cea6bf4880d60 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4996 0x04881d00 0x0000ea6bf4881d00 0x166cea6bf4881d00 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857f80 0x04883470 0x0000ea6bf4883470 0x166cea6bf4883470 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f50 0x04883c40 0x0000ea6bf4883c40 0x166cea6bf4883c40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e3b 0x04884410 0x0000ea6bf4884410 0x166cea6bf4884410 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f31 0x048853b0 0x0000ea6bf48853b0 0x166cea6bf48853b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7db2 0x04885b80 0x0000ea6bf4885b80 0x166cea6bf4885b80 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4951 0x04886b20 0x0000ea6bf4886b20 0x166cea6bf4886b20 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27e54 0x0488a1d0 0x0000ea6bf488a1d0 0x166cea6bf488a1d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bc5 0x0488b170 0x0000ea6bf488b170 0x166cea6bf488b170 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947d43 0x0488d0b0 0x0000ea6bf488d0b0 0x166cea6bf488d0b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f7a 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e13 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4985 0x0488ff90 0x0000ea6bf488ff90 0x166cea6bf488ff90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7f0e 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dc7 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4db1 0x04893e10 0x0000ea6bf4893e10 0x166cea6bf4893e10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ea2 0x07d8b160 0x0000ea6d67d8b160 0x166cea6d67d8b160 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7eda 0x07d8b930 0x0000ea6d67d8b930 0x166cea6d67d8b930 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7c0e 0x07d8d0a0 0x0000ea6d67d8d0a0 0x166cea6d67d8d0a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bdb 0x07d8efe0 0x0000ea6d67d8efe0 0x166cea6d67d8efe0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947de3 0x07d90750 0x0000ea6d67d90750 0x166cea6d67d90750 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fed 0x07d945d0 0x0000ea6d67d945d0 0x166cea6d67d945d0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4f9a 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07f5e 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7e2b 0x07d95570 0x0000ea6d67d95570 0x166cea6d67d95570 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c6a 0x07d98c20 0x0000ea6d67d98c20 0x166cea6d67d98c20 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f27 0x07d99bc0 0x0000ea6d67d99bc0 0x166cea6d67d99bc0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7d09 0x07d9a390 0x0000ea6d67d9a390 0x166cea6d67d9a390 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937d1a 0x07d9bb00 0x0000ea6d67d9bb00 0x166cea6d67d9bb00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e345 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0878187 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ce8 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c38a0 0x07d9e9e0 0x0000ea6d67d9e9e0 0x166cea6d67d9e9e0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4b25 0x07da2090 0x0000ea6d67da2090 0x166cea6d67da2090 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c55 0x07da3030 0x0000ea6d67da3030 0x166cea6d67da3030 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937e6c 0x07da47a0 0x0000ea6d67da47a0 0x166cea6d67da47a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c36aa 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07c8e 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837f38 0x07da9d90 0x0000ea6d67da9d90 0x166cea6d67da9d90 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0917d84 0x07daad30 0x0000ea6d67daad30 0x166cea6d67daad30 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f39 0x07dab500 0x0000ea6d67dab500 0x166cea6d67dab500 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d15 0x07dabcd0 0x0000ea6d67dabcd0 0x166cea6d67dabcd0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d6c 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7f77 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e346 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937f84 0x07dadc10 0x0000ea6d67dadc10 0x166cea6d67dadc10 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bb4 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07cee 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e94 0x07db3200 0x0000ea6d67db3200 0x166cea6d67db3200 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d16 0x07db5910 0x0000ea6d67db5910 0x166cea6d67db5910 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937dd1 0x07db7080 0x0000ea6d67db7080 0x166cea6d67db7080 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7e42 0x07db87f0 0x0000ea6d67db87f0 0x166cea6d67db87f0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ed0 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e15 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f69 0x07dbaf00 0x0000ea6d67dbaf00 0x166cea6d67dbaf00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e60 0x07dbc670 0x0000ea6d67dbc670 0x166cea6d67dbc670 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bfb 0x07dbed80 0x0000ea6d67dbed80 0x166cea6d67dbed80 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837eb4 0x055a8c50 0x0000ea6d79a8c844 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7dba 0x055a8c50 0x0000ea6d755a8c50 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e6d 0x055a9420 0x0000ea6d755a9420 0x166cea6d755a9420 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0907bef 0x055aa3c0 0x0000ea6d755aa3c0 0x166cea6d755aa3c0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7ed5 0x055acad0 0x0000ea6d755acad0 0x166cea6d755acad0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0887eb6 0x055ae240 0x0000ea6d755ae240 0x166cea6d755ae240 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4da2 0x055aea10 0x0000ea6d755aea10 0x166cea6d755aea10 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ff1 0x055b0180 0x0000ea6d755b0180 0x166cea6d755b0180 0x166cea6d79a8c844

 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0xc0da81a0 0x0bb28852 0x0000ee992bb28852 0x166cee992bb28852 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f4811f 0x0bb29022 0x0000ee992bb29022 0x166cee992bb29022 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc805b 0x0bb297f2 0x0000ee992bb297f2 0x166cee992bb297f2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d38174 0x0bb2af62 0x0000ee992bb2af62 0x166cee992bb2af62 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38233 0x0bb2ede2 0x0000ee992bb2ede2 0x166cee992bb2ede2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68019 0x0bb2f5b2 0x0000ee992bb2f5b2 0x166cee992bb2f5b2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0x83807863 0x0bb30fca 0x0000ee992bb30fca 0x166cee992bb30fca 0x166cee992d6fc8b2
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f3c 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7bc6 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e12 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7df4 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770

Earliest & latest timestamps from latest batch = Tuesday, March 16, 2021 7:48:18.767 PM - Tuesday, March 16, 2021 9:06:38.184 PM this evening
 
21.17 FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.5M data items/s
      TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-16_Statistics_aidas-gsi.png
Screenshot_2021-03-16_Statistics_aidas-gsi.png
Attachment 3: 82.png
82.png
Attachment 4: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-16_Statistics_aidas-gsi(1).png
Screenshot_2021-03-16_Statistics_aidas-gsi(1).png
Attachment 6: 83.png
83.png
  201   Mon Mar 15 11:47:51 2021 TDMonday 15 March
12.48 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

12.50 System wide checks

	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20     18      5      4      2      2      2      2      3      4     10   : 54576
aida02 :     26     18     16     10      6      4      2      4      2      3      6   : 37048
aida03 :     18     10      4      3      2      3      2      4      2      3      6   : 36152
aida04 :     13      9      4      2      2      2      2      4      1      4     15   : 73852
aida05 :     46     72     70     54     52     48     21     19      7      4      7   : 72216
aida06 :     32     10      4     10      0      2      2      4      3      2      5   : 31056
aida07 :     19      7      5      3      2      3      3      3      3      3      6   : 36916
aida08 :     20     14      2      3      1      3      1      4      3      3      6   : 36864
aida09 :      3      6      3      3      2      4      1      2      3      3      6   : 35916
aida10 :     23     11      3      3      1      3      3      2      3      3      6   : 36356
aida11 :      2      5      6      2      2      4      2      3      2      3      6   : 35664
aida12 :     25     10      5      2      2      2      2      3      3      3      6   : 36548


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

12.55 Rate spectra - attachments 4 & 5

      Merger server error messages since most recent check

MERGE Data Link (20503): bad timestamp  3 3 0xc0d582ee 0x04f02352 0x000064aa97efd660 0x166c64aa94f02352 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e680cf 0x04f032f2 0x000064aa94f032f2 0x166c64aa94f032f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68215 0x04f03ac2 0x000064aa94f03ac2 0x166c64aa94f03ac2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd81f1 0x04f061d2 0x000064aa94f061d2 0x166c64aa94f061d2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb812a 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68041 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c183d5 0x04f0aff2 0x000064aa94f0aff2 0x166c64aa94f0aff2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d58220 0x04f0b7c2 0x000064aa94f0b7c2 0x166c64aa94f0b7c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68097 0x04f0bf92 0x000064aa94f0bf92 0x166c64aa94f0bf92 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c682e7 0x04f0c762 0x000064aa94f0c762 0x166c64aa94f0c762 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf7ff7 0x04f0cf32 0x000064aa94f0cf32 0x166c64aa94f0cf32 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd817e 0x04f0ee72 0x000064aa94f0ee72 0x166c64aa94f0ee72 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e180e5 0x04f105e2 0x000064aa94f105e2 0x166c64aa94f105e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181fd 0x04f10db2 0x000064aa94f10db2 0x166c64aa94f10db2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d78072 0x04f11582 0x000064aa94f11582 0x166c64aa94f11582 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c17fff 0x04f14462 0x000064aa94f14462 0x166c64aa94f14462 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68234 0x04f15bd2 0x000064aa94f15bd2 0x166c64aa94f15bd2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780f9 0x04f17342 0x000064aa94f17342 0x166c64aa94f17342 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98277 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28174 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d5812d 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd8112 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d98224 0x04f18ab2 0x000064aa94f18ab2 0x166c64aa94f18ab2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68096 0x04f19a52 0x000064aa94f19a52 0x166c64aa94f19a52 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c080e1 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d18282 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38198 0x04f1a9f2 0x000064aa94f1a9f2 0x166c64aa94f1a9f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cb8187 0x04f1b1c2 0x000064aa94f1b1c2 0x166c64aa94f1b1c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8178 0x04f1b992 0x000064aa94f1b992 0x166c64aa94f1b992 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c78063 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e980e1 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98129 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28107 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb80e2 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660

18.47 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

      No new merger error messages since 12.55 today

          
Attachment 1: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-15_Statistics_aidas-gsi.png
Screenshot_2021-03-15_Statistics_aidas-gsi.png
Attachment 3: 60.png
60.png
Attachment 4: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 7: Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Attachment 8: 61.png
61.png
  200   Sun Mar 14 17:35:05 2021 TD, OHSunday 14 March - Pulser Walkthrough
18.01 Pulser walkthrough at end of run S452 file S452/R57_0 ... R57_28

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      BNC PB-5 settings changed by terminal command line (see attachments 1, 2 & 3)
      using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      Example n+n LEC spectrum - see attachment 4

18.27 R57 ends OK file R57_28
Attachment 1: 52.png
52.png
Attachment 2: 51.png
51.png
Attachment 3: 40.png
40.png
Attachment 4: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
  199   Sun Mar 14 08:16:00 2021 OH PJCSSunday 14th March 2021
> 08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
> 
> Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
> It is also the least noisy FEE in the back DSSD
> 
> MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
> MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
> 
> 09:01 System wide checks
> 
> 		 Base 		Current 	Difference
> aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
> White Rabbit error counter test result: Passed 11, Failed 1
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 			 Base 		Current 		Difference
> aida07 fault 	 0x2 : 	 0x5 : 	 3  
> FPGA Timestamp error counter test result: Passed 11, Failed 1
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
>      Statistics - attachment 1
>      Temperature - attachment 2
>      Bias and leakage current - attachment 3
It might be worth considering decreasing the time between the flushing of slower FEEs. Is the Merger waiting for the slower data and causing ‘deadtime’ in the faster ones. Perhaps consider this as a 
balancing across the system? 
  198   Sun Mar 14 07:04:30 2021 OHSunday 14th March 2021
08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151

Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
It is also the least noisy FEE in the back DSSD

MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262

09:01 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Statistics - attachment 1
     Temperature - attachment 2
     Bias and leakage current - attachment 3

11:18 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - attachment 4
     Temperatures - attachment 5
     Bias and leakage - attachment 6


13.14 DAQ continues OK - file R56_285

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

13.18 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      8      3      4      2      2      2      2      3      4     10   : 54492
aida02 :      0      2      3      0      2      3      2      3      3      3      6   : 36416
aida03 :     18      9      1      2      0      4      1      3      3      3      6   : 36320
aida04 :      2      3      3      3      1      4      3      2      2      4     15   : 74224
aida05 :     26     72     72     54     53     49     22     18      7      4      7   : 72104
aida06 :      7      5      5     10      0      2      2      4      3      2      5   : 30932
aida07 :      2      4      2      1      1      3      2      3      2      3      7   : 39464
aida08 :     25      9      4      2      1      2      1      4      3      3      6   : 36716
aida09 :      4     11      3      5      2      4      1      2      3      3      6   : 36024
aida10 :     18      5      2      1      0      5      2      2      3      3      6   : 36144
aida11 :     25     10      4      1      1      4      2      3      2      3      6   : 35668
aida12 :     26      7      1      1      1      3      2      3      3      3      6   : 36496

13.19 FEE64 temperatures OK - attachment 7
      Good events stats OK - attachment 8
      Bias & leakage currents OK - attachment 9
      Merger 4.5M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      All histograms zero'd


16.00 DAQ contiues R56_361

       System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c79 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      8      6      4      2      2      2      2      3      4     10   : 54524
aida02 :      1      3      4      2      1      2      1      4      3      3      6   : 36572
aida03 :      2      3      2      3      1      3      2      2      3      3      6   : 35936
aida04 :     19      7      2      3      1      4      3      3      1      4     15   : 73796
aida05 :     34     68     72     54     53     49     22     18      7      4      7   : 72104
aida06 :     11     10      4     10      0      2      2      4      3      2      5   : 30972
aida07 :      1      3      2      2      2      2      2      3      2      3      7   : 39420
aida08 :     24     10      5      2      1      3      1      4      3      3      6   : 36864
aida09 :     20      8      3      5      2      3      2      3      3      3      6   : 36704
aida10 :      2      3      2      3      1      3      3      2      3      3      6   : 36192
aida11 :     17     10      4      0      1      4      2      3      2      3      6   : 35604
aida12 :     20      6      3      2      1      3      2      2      2      4      6   : 37040


15.05 FEE64 temperatures OK - attachment 10
      Good events stats OK - attachment 11
      Bias & leakage currents OK - attachment 12
      Merger 4.6M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      Rate spectra - attachments 13 & 14
      p+n junction HEC spectra - attachment 15

17.33 end of S452 run file R56_397
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      BNC PB-5 config - attachment 16
      CAEN 8xxx NIM bin config - attachment 17
Attachment 1: 210314_0858_Stats.png
210314_0858_Stats.png
Attachment 2: 210314_0859_Temp.png
210314_0859_Temp.png
Attachment 3: 210314_0900_Bias.png
210314_0900_Bias.png
Attachment 4: 210314_1116_Stats.png
210314_1116_Stats.png
Attachment 5: 210314_1116_Temp.png
210314_1116_Temp.png
Attachment 6: 210314_1117_Bias.png
210314_1117_Bias.png
Attachment 7: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi.png
Attachment 8: Screenshot_2021-03-14_Statistics_aidas-gsi.png
Screenshot_2021-03-14_Statistics_aidas-gsi.png
Attachment 9: 34.png
34.png
Attachment 10: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 11: Screenshot_2021-03-14_Statistics_aidas-gsi(1).png
Screenshot_2021-03-14_Statistics_aidas-gsi(1).png
Attachment 12: 35.png
35.png
Attachment 13: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png
Attachment 15: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 16: 40.png
40.png
Attachment 17: 41.png
41.png
Attachment 18: 51.png
51.png
Attachment 19: 52.png
52.png
Attachment 20: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
  197   Sat Mar 13 23:08:08 2021 CA14th March 00:00-08:00
00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC
Attachment 1: Screenshot_from_2021-03-13_23-14-33.png
Screenshot_from_2021-03-13_23-14-33.png
Attachment 2: Screenshot_from_2021-03-13_23-14-57.png
Screenshot_from_2021-03-13_23-14-57.png
Attachment 3: Screenshot_from_2021-03-13_23-15-12.png
Screenshot_from_2021-03-13_23-15-12.png
Attachment 4: Screenshot_from_2021-03-14_01-22-56.png
Screenshot_from_2021-03-14_01-22-56.png
Attachment 5: Screenshot_from_2021-03-14_01-29-35.png
Screenshot_from_2021-03-14_01-29-35.png
Attachment 6: Screenshot_from_2021-03-14_01-29-52.png
Screenshot_from_2021-03-14_01-29-52.png
Attachment 7: Screenshot_from_2021-03-14_01-40-57.png
Screenshot_from_2021-03-14_01-40-57.png
Attachment 8: Screenshot_from_2021-03-14_02-16-28.png
Screenshot_from_2021-03-14_02-16-28.png
Attachment 9: Screenshot_from_2021-03-14_02-17-01.png
Screenshot_from_2021-03-14_02-17-01.png
Attachment 10: Screenshot_from_2021-03-14_02-17-32.png
Screenshot_from_2021-03-14_02-17-32.png
Attachment 11: Screenshot_from_2021-03-14_02-29-29.png
Screenshot_from_2021-03-14_02-29-29.png
Attachment 12: R56_35
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261409816 ( 1907248.6 Hz)
        Other data format:     510184 (    3722.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2488 (      18.2 Hz)
                               RESUME:       2487 (      18.1 Hz)
                              SYNC100:        511 (       3.7 Hz)
                              WR48-63:        511 (       3.7 Hz)
                           FEE64 disc:     503410 (    3672.9 Hz)
                             MBS info:        777 (       5.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      60911 (     444.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.061 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.725              132.339
  2                3.132              135.291
  3                6.180              132.876
  4               12.844              128.581
  5                8.386              132.070
  6                1.216              132.607
  7                1.093              115.159
  8               50.447              129.654
  9               19.940              134.755
 10                0.568              132.339
 11               51.844              133.681
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22917695      26526          0          0        878        878         65         65      23863        777          0       8590
  1   26593258      35084          0          0         13         13         51         51      34956          0          0      10875
  2   25634005     337148          0          0         16         16         76         76     336964          0          0       6926
  3   28548879       3343          0          0         34         34         27         27       3221          0          0       1214
  4    7607641      18967          0          0        663        663         14         14      17613          0          0       6303
  5   19366719      25239          0          0        305        304         24         24      24582          0          0       8037
  6   18601227      17453          0          0          3          3         18         18      17411          0          0       6563
  7   10076045       2592          0          0          4          4         24         24       2536          0          0        908
  8   30026924       8042          0          0        217        216         34         34       7541          0          0       2274
  9   30858440      18084          0          0        113        113         88         88      17682          0          0       4590
 10    9946552      15820          0          0          3          3         46         46      15722          0          0       4264
 11   31232431       1886          0          0        239        240         44         44       1319          0          0        367
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:13611.407s (    2.351 blocks/s,   0.147 Mb/s)
Attachment 13: Screenshot_from_2021-03-14_04-20-07.png
Screenshot_from_2021-03-14_04-20-07.png
Attachment 14: Screenshot_from_2021-03-14_04-20-34.png
Screenshot_from_2021-03-14_04-20-34.png
Attachment 15: Screenshot_from_2021-03-14_04-20-50.png
Screenshot_from_2021-03-14_04-20-50.png
Attachment 16: Screenshot_from_2021-03-14_04-25-41.png
Screenshot_from_2021-03-14_04-25-41.png
Attachment 17: Screenshot_from_2021-03-14_04-32-03.png
Screenshot_from_2021-03-14_04-32-03.png
Attachment 18: Screenshot_from_2021-03-14_04-32-32.png
Screenshot_from_2021-03-14_04-32-32.png
Attachment 19: Screenshot_from_2021-03-14_04-33-14.png
Screenshot_from_2021-03-14_04-33-14.png
Attachment 20: Screenshot_from_2021-03-14_06-10-39.png
Screenshot_from_2021-03-14_06-10-39.png
Attachment 21: Screenshot_from_2021-03-14_06-11-19.png
Screenshot_from_2021-03-14_06-11-19.png
Attachment 22: Screenshot_from_2021-03-14_06-11-38_-_1.png
Screenshot_from_2021-03-14_06-11-38_-_1.png
  196   Sat Mar 13 17:00:49 2021 TDSaturday 13 March 18.00-00.00 UTC+1
18.00 DAQ continues OK - file R46_64

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

18.02 System wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

17.15 Most recent messages in merger server terminal session

MERGE Data Link (2322): bad timestamp  6 3 0xc1b57e35 0x0ba1685c 0x0000ef3fbba1685c 0x166bef3fbba1685c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a77dc2 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bf7f66 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
Transfer Error - : Broken pipe
Error 32
1: send() failed:
TCP transfer library version 4.0
1: TCP socket send buffer was 16384 - now 249856
1: TCP socket receive buffer was 87380 - now 249856
1: TCP socket created OK - now connecting to localhost port 10305
1: Connected to localhost port 10305


22.05 DAQ continues OK - file R46_168

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a2 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18      6      7      5      1      3      1      3      3      4     10   : 54856
aida02 :     19      6      4      3      4      4      2      3      3      3      6   : 36892
aida03 :     22      8      1      2      1      2      1      3      3      3      6   : 36136
aida04 :     24      8      4      2      1      3      2      4      1      4     15   : 73952
aida05 :     37     14      9      8      7      3      2     18      7      4      7   : 55252
aida06 :     28     12     16      4      4      2      1      4      3      2      5   : 31056
aida07 :     28      9      2      2      2      2      1      3      3      3      6   : 36248
aida08 :     21     12      6      4      3      4      2      3      3      3      6   : 36948
aida09 :     18      5      2      1      1      3      2      2      3      3      6   : 35952
aida10 :     17      8      4      5      3      3      1      3      3      3      6   : 36516
aida11 :     21      8      5      0      0      4      3      3      2      3      6   : 35812
aida12 :     13     11      3      3    


      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.4M data items/s
      TapeServer OK - 15Mb/s

22.17 Rate spectra - attachments 7 & 8
      p+n junction HEC spectra - attachment 9

      Merger server error messages since 17.15

MERGE Data Link (2316): bad timestamp  0 3 0x80300000 0x0164e7f8 0x000000000164e7f8 0x166200000164e7f8 0x166bffffffa7654e
MERGE Data Link (2324): bad timestamp  8 3 0x88300000 0x0534e25c 0x000000000534e25c 0x166200000534e25c 0x166bffffed8383fc
MERGE Data Link (2322): bad timestamp  6 3 0xc1a07f16 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b37895 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc19f7e41 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a27e22 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b47399 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ab7dad 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b564f9 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b60177 0x0071013c 0x000001e74071013c 0x166c01e74071013c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a57e95 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b81491 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ac7e4f 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b90000 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ba0c64 0x00e2302c 0x000001e700e2302c 0x166c01e700e2302c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bb6ea8 0x00e237fc 0x000001e700e237fc 0x166c01e700e237fc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a87dc6 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bc0641 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
Attachment 1: Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-13_Statistics_aidas-gsi.png
Screenshot_2021-03-13_Statistics_aidas-gsi.png
Attachment 3: 32.png
32.png
Attachment 4: Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-13_Statistics_aidas-gsi(1).png
Screenshot_2021-03-13_Statistics_aidas-gsi(1).png
Attachment 6: 33.png
33.png
Attachment 7: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(2).png
  195   Sat Mar 13 07:08:52 2021 CA, TD, LSMarch 13th 08:00 - 17:00
ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:09 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

08:14 FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias/leakage current ok - attachment 3

08:22 Merger 4.3M items/s

      Tapeserver 14 MB/s

08:24 DAQ ok, writing to file R43_45

08:39 rate spectra - attachment 4

      aida09 spectrum not showing, however continues to collect statistics ok

10:03 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

      FEE64 temperatures ok - attachment 5

      good event statistics ok - attachment 6

      detector bias/leakage current ok - attachment 7

10:10 Merger 4.5M items/s

      Tapeserver 14 MB/s

      data forwarding to MBS ok

10:11 writing to file R43_91

12.00 (LS)
      System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      		         Base 		Current 	Difference
      aida07 fault 	 0x829e : 	 0x829f : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1
      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      no FGPA timestamp errors all passed

      Statistics (attachment8)
      Spectra rate (attachment9), aida09 still does not show histogram, but on run control are enabled for all fees. still 
      showing good events and other information (temperatures etc)
      FEE temps (attachment10)
      Leakage currents written to sheets (attachment11)
      Merger~4.6M items/s
      Tapeserver~14MB/s

12.36 bunch of bad timestamp errors in the new merger terminal (attachment 12) around file R43_152
      this error is still occurring at 12.45 will continue to monitor, all other system checks are reporting as normal


13.50 beam down file R43_183
      beam back a couple of minutes later

14.00 System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      no white rabbit or FPGA errors all passed

      Statistics (attachment13)
      Spectra rate (attachment14)
      FEE temps (attachment15)
      Leakage currents written to sheets (attachment16)
      Merger~4.6M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay, still seeing bad timestamp errors in the new merger terminal continuing to monitor and update 
      statistics tab every 20 to 30 minutes 

14.12 Analysed files R43_185,186,187 from timestamp error in ucesb, but see no timewarps  

14.30 have not seen any bad timestamp errors in new merger terminal for a while, unsure if something has been changed which 
      has stopped them

15.25 beam down
15.53 beam back

16.00 moved back to writing to /media/SecondDrive which should last until the end of the experiment
      now writing R46

16.10 System wide checks okay except:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment17)
      Spectra rate (attachment18)
      FEE temps (attachment19)
      Leakage currents written to sheets (attachment20)
      Merger~4.7M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay
Attachment 1: Screenshot_from_2021-03-13_07-12-39.png
Screenshot_from_2021-03-13_07-12-39.png
Attachment 2: Screenshot_from_2021-03-13_07-13-15.png
Screenshot_from_2021-03-13_07-13-15.png
Attachment 3: Screenshot_from_2021-03-13_07-13-46.png
Screenshot_from_2021-03-13_07-13-46.png
Attachment 4: Screenshot_from_2021-03-13_07-38-52.png
Screenshot_from_2021-03-13_07-38-52.png
Attachment 5: Screenshot_from_2021-03-13_09-04-04.png
Screenshot_from_2021-03-13_09-04-04.png
Attachment 6: Screenshot_from_2021-03-13_09-04-28.png
Screenshot_from_2021-03-13_09-04-28.png
Attachment 7: Screenshot_from_2021-03-13_09-04-50.png
Screenshot_from_2021-03-13_09-04-50.png
Attachment 8: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 11: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 12: 1200_NewMerger.PNG
1200_NewMerger.PNG
Attachment 13: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 14: 1400_SpectraRate.PNG
1400_SpectraRate.PNG
Attachment 15: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 16: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 17: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 18: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 19: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 20: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
  194   Fri Mar 12 23:56:00 2021 OHSaturday March 13th 00:00-08:00
24:00 Joined in with TD to debug the dropout of AIDA05 - See https://elog.ph.ed.ac.uk/DESPEC/193
      After both powercycles and telnet reboots we were still seeing 0 in the statistics but did note that the counter was > 0
      The last check we did was a restart of the merger server. This solved the issue.
      It is likely that the link to the FEE was dropped but not re-established upon the resets.
      There was no error message that this was seen. From now on I recommend refreshing the statistics ever 30 minutes.

01:29 System wide check
N.B. May not have reset baselines after reset
WR fault
		 Base 		Current 	Difference
aida01 fault 	 0x1ad : 	 0x1af : 	 2  
aida02 fault 	 0x4434 : 	 0x4436 : 	 2  
aida03 fault 	 0xcf95 : 	 0xcf99 : 	 4  
aida04 fault 	 0x25aa : 	 0x25ae : 	 4  
aida05 fault 	 0x4b4 : 	 0x4b7 : 	 3  
aida06 fault 	 0x834 : 	 0x837 : 	 3  
aida07 fault 	 0x146d : 	 0x1472 : 	 5  
aida08 fault 	 0xf4a8 : 	 0xf4ab : 	 3  
aida09 fault 	 0x4f6f : 	 0x4f73 : 	 4  
aida10 fault 	 0x7504 : 	 0x7506 : 	 2  
aida11 fault 	 0x26fa : 	 0x26fc : 	 2  
aida12 fault 	 0x2858 : 	 0x285b : 	 3  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Check
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

01:42 Statistics - attachment 1
      Temperature - attachment 2
      Bias and leakage current - attachment 3

03:57 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1473 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x4 : 	 0xc : 	 8  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics - attachment 4
      Temp - attachment 5
      Bias - attachment 6

05:27 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1474 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x4 : 	 0x16 : 	 18  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


     Statistics - attachment 7
     Temp - attachment 8
     Bias - attachment 9

05:51 Looking into the bad timestamp messages in the merger e.g.
MERGE Data Link (28348): bad timestamp  5 3 0xc14f8415 0x00eea280 0x0000cd2e70eea280 0x166bcd2e70eea280 0x166bcd2e716c1f80

Looking in the merger source for the bad timestamp message:

            if (op->Time < LastTimeStamp) {

//     invalid time stamp 

               (*StatsMem[TSSEQERR])++;

               (*StatsMem[TSSEQERR+((LinkNum+1)*MAXCOUNTERS)])++;

               sprintf(message_buffer, "bad timestamp  %d %d 0x%08lx 0x%08lx 0x%016llx 0x%016llx 0x%016llx",LinkNum, link_table[LinkNum]->link_state, op->Data, op->Timestamp, INFO4, op->Time, LastTimeStamp);
      
                report_message(MSG_WARNING);   /***************************/

//              LastTimeStamp = op->Time;


So the message is generated when the merger detects a timewarp.

Took the first warning a data block of errors (The first instance of that particular `LastTimeStamp` and c alculated the time difference between the new timestamp and the last timestamp
The time and LastTimestamp were 0x166bccef3f424a50 0x166bccef4f4232e0 respectively
The time difference between them is 268429456 which is 268ms which seems quite a large difference


06:39 AIDA Crashed
      This time all FEEs were responsive but not showing any stats
      When stopping the DAQ all FEEs except aida06 stopped - attachment 10
      Did a reset of the DAQ and all recovered but no stats on aida06 - attachment 11
      Regained DAQ with a powercycle and a complete restart of the AIDA:8115 Merger and TaperServer

      It is worth noting that aida06 is connected to link 5 the data link which had been producing the bad merge messages overnight.
      We have now had it in aida05, aida6 and aida07. Could it be to do with the correlation scaler rate going into these FEES?

      Going through the var/log/messages on aida-3 aida06 rebooted itself at 06:37
      Mar 13 06:37:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Mar 13 06:37:18 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Mar 13 06:37:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

Looking in /var/log/messages on aida06 no evidence of a reason why:
Mar 12 23:30:56 aida06 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: driver assigned major number 253
Mar 12 23:31:13 aida06 kernel: xaida: open:
Mar 12 23:31:14 aida06 kernel: AIDAMEM: aidamem_open:
Mar 12 23:33:06 aida06 kernel: xaida: open:
Mar 13 05:37:30 aida06 syslogd 1.4.2: restart.
Mar 13 05:37:30 aida06 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Mar 13 05:37:30 aida06 kernel: Using Xilinx Virtex440 machine description
Mar 13 05:37:30 aida06 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Mar 13 05:37:30 aida06 kernel: Zone PFN ranges:
Attachment 1: 210313_0138_Stats.png
210313_0138_Stats.png
Attachment 2: 210313_0141_Temp.png
210313_0141_Temp.png
Attachment 3: 210313_0142_Bias.png
210313_0142_Bias.png
Attachment 4: 210313_0355_Stats.png
210313_0355_Stats.png
Attachment 5: 210313_0356_Bias.png
210313_0356_Bias.png
Attachment 6: 210313_0356_Temp.png
210313_0356_Temp.png
Attachment 7: 210313_0528_Stats.png
210313_0528_Stats.png
Attachment 8: 210313_0529_Temp.png
210313_0529_Temp.png
Attachment 9: 210313_0530_Bias.png
210313_0530_Bias.png
Attachment 10: 2210313_0640_AIDAissue.png
2210313_0640_AIDAissue.png
Attachment 11: 210313_0647_aida6MMissing.png
210313_0647_aida6MMissing.png
  193   Fri Mar 12 20:12:10 2021 TDFriday 12 March
21.12 DAQ continues OK - file R33_617

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

21.15 System wide checks


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a58 : 	 0x7a5a : 	 2  
aida06 fault 	 0x65bd : 	 0x65bf : 	 2  
aida07 fault 	 0xcdd6 : 	 0xcdda : 	 4  
aida08 fault 	 0x2ab5 : 	 0x2ab7 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      4      2      0      1      2      3      3      2      3     11   : 55956
aida02 :     21      4      1      2      2      4      2      2      1      4      7   : 40260
aida03 :     10      4      4      1      6      4      1      3      3      3      6   : 36648
aida04 :      7      2      4      0      2      3      2      4      1      4     15   : 73836
aida05 :      1      3      1      5      4      3      1      2      3      4      6   : 37964
aida06 :     12      3      5      2      1      3      3      3      2      4      7   : 41880
aida07 :      2      4      3      3      4      2      2      4      3      3      6   : 37048
aida08 :     17     13      2      3      2      3      2      3      2      3      7   : 39724
aida09 :      3      7      2      0      1      2      2      3      2      2      7   : 37284
aida10 :      2      5      3      2      2      3      1      3      2      3      7   : 39328
aida11 :     23     18      2      3      2      3      3      2      3      3      6   : 36460
aida12 :     18      7      3      1      1      4      2      2      2      3      7   : 39184

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.9M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

21.50 At some point between R33_610 (20.59) and R33_625 (21.30) aida05 stopped producing data (zero good events - see attachment 2)

      able to telnet to aida05 - no warnings/error messages in /var/log/messages

      DAQ STOP (all except aida05 stopped OK, aida05 remained GOING)

      issued aida05 reboot command via telnet command line

      DAQ RESET/SETUP/GO (all FEE64s GOING OK except aida05 - zero good events)

21.15 All system wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


21.50 aida04 & aida05 statistics for comparison - see attachments 4 & 5
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Attachment 3: 31.png
31.png
Attachment 4: Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
  192   Fri Mar 12 10:09:19 2021 LS, CAFriday 12th March 11.00-
11.00(Germany) System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment1)
      Spectra rates (attachment2)
      FEE temps (attachment3)
      Leakage currents, written to google sheet (attachment4)  
      Merger~ 4.9M items/s
      Tapeserver ~17MB/s 

      In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
      AIDA has been taken out of the timesorter due to the high data rate, buffers were full
      AIDA cannot be seen in ucesb or Go4.

13.00 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment5)
      Spectra rates (attachment6)
      FEE temps (attachment7)
      Leakage currents, written to google sheet (attachment8)  
      Merger~ 5.1M items/s
      Tapeserver ~18MB/s 

      No timestamp related errors this shift

13.37 AIDA MBS control restarted R33_388
      System wide checks same as previous time

13.54 beam stopped for access,  file R33_396

13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
      Analysed R33_395, 396, 397, 398 no timewarps
      
14.29 beam back R33_415

14.54 see more bad timestamps in new merger terminal (attachment10)

14.55 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment11)
      Spectra rates (attachment12)
      FEE temps (attachment13)
      Leakage currents, written to google sheet (attachment14)  
      Merger~ 5.0M items/s
      Tapeserver ~17MB/s 

16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463

16.15 CA takes over until 18:00

17:11 System wide checks:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0xcdd6 : 	 0xcdd7 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

17:13 FEE64 temps ok - attachment 15
      Statistics ok - attachment 16
      bias and leakage currents ok - attachment 17
Attachment 1: 1100_Statistics.PNG
1100_Statistics.PNG
Attachment 2: 1100_SpectraRate.PNG
1100_SpectraRate.PNG
Attachment 3: 1100_FEETemps.PNG
1100_FEETemps.PNG
Attachment 4: 1100_LeakageCurrents.PNG
1100_LeakageCurrents.PNG
Attachment 5: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 6: 1300_SpectraRate.PNG
1300_SpectraRate.PNG
Attachment 7: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 8: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 9: 1358_NewMerger.PNG
1358_NewMerger.PNG
Attachment 10: 1454_NewMerger.PNG
1454_NewMerger.PNG
Attachment 11: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 12: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 13: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 14: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 15: Screenshot_from_2021-03-12_16-18-13.png
Screenshot_from_2021-03-12_16-18-13.png
Attachment 16: Screenshot_from_2021-03-12_16-18-51.png
Screenshot_from_2021-03-12_16-18-51.png
Attachment 17: Screenshot_from_2021-03-12_16-21-21.png
Screenshot_from_2021-03-12_16-21-21.png
  191   Fri Mar 12 08:32:46 2021 TDFriday 12 March 08.00-
08.31 DAQ continues OK - file R33_263

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08.32 No merger server error/warning messages since most recent restart c. 00.00 today


08.44 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     14      6      3      2      1      2      2      3      3      4     11   : 58904
aida02 :      6      4      1      1      1      2      2      4      2      3      7   : 39848
aida03 :      5      6      4      2      1      3      1      3      2      3      7   : 39300
aida04 :      4      1      2      2      3      3      2      4      1      4     15   : 73912
aida05 :      1      5      2      2      2      4      2      3      2      3      7   : 39692
aida06 :      7      2      2      3      1      3      3      3      2      4      7   : 41836
aida07 :      6      3      1      2      3      2      2      3      3      3      7   : 40512
aida08 :      8      5      2      2      1      3      1      3      2      3      7   : 39272
aida09 :      1      2      3      1      1      3      2      3      1      4      7   : 40484
aida10 :      2      6      3      2      1      2      2      2      3      3      7   : 39912
aida11 :      2      3      1      0      2      4      1      3      2      3      7   : 39344
aida12 :      2      3      4      2      0      3      3      3      2      3      7   : 39712


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 5.1M data items/s
      TapeServer OK - 16Mb/s

08.54 p+n junction strip HEC spectra - attachment 1
      Rate spectra - attachment 2

09.15 Burst of Merger server warning messages of type

MERGE Data Link (3547): bad timestamp  1 3 0x8128d6c8 0x05056328 0x00008d6c85056328 0x166b8d6c85056328 0x166b8d6d86da01ee

      also multiple

Warning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one 
MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in 
relayWarning: At least one MIDAS block missed in relayWarning: A

      The latter may reflect downstream problems writing MBS data
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi.png
Screenshot_2021-03-12_Statistics_aidas-gsi.png
Attachment 3: 30.png
30.png
Attachment 4: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
  190   Thu Mar 11 23:07:23 2021 CAMarch 12th 00:00 - 08:00 shift
00:10 DAQ continues OK - file R30_372

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:17 System wide checks all OK

      FEE64 Temperatures ok - attachment 1

      Good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3

00:20 DESPEC on run 145

00:24 Merger ok ~45M items/s
      Tapeserver ok ~15MB/s

01:00 aida07 crashed

      recovered, but all FEE64 crash shortly after

      called OH - stop DAQ, TapeService, Merger

01:20 all FEE64 powercycled, AIDA restart

01:30 AIDA recovered, DAQ now running - writing to R33

      system wide checks ok *except*

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

01:52 FEE64 Temperatures ok - attachment 4

      -had to reload a few times to work, but otherwise ok

      Good event statistics - attachment 5

      -aida05 and aida08 running faster than before

      detector bias and leakage currents ok - attachment 6

02:04 writing to file R33_34

      Data forwarding to MBS ok

      AIDA ASIC settings ok

02:11 beam off

02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails 

02:32 beam back - writing to file R33_48

04:30 DESPEC having issues with Go4 crashing

      ucesb reports AIDA/FATIMA/bplast timewarp events

      System wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
       Base 		Current 	Difference
       aida07 fault 	 0xcdd1 : 	 0xcdd2 : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temperatures ok - attachment 7

      Good event statistics ok - attachment 8

      detector bias and leakage currents ok - attachment 9

04:52 analyzer output for R33_113 - attachment 10

      no timewarps

      aida06 dead time? (ignore idle time and rates)

05:02 Merger 5M data items/s

      TapeServer 17 MB/s

05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash

      They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now

05:25 rates spectra - attachment 11

05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12

06:29 system wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
                         Base 		Current 	Difference
      aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
      aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
      aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
      aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
      White Rabbit error counter test result: Passed 8, Failed 4

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      rest ok

06:32 FEE64 Temperatures ok - attachment 13

      Good event statistics ok - attachment 14

      detector bias and leakage currents ok - attachment 15

06:41 Merger 5.1M data items/s

      TapeServer 17 MB/s

      Data forwarding to MBS ok

08:54 restart MBS relay, requested by NH

      













      

      
Attachment 1: Screenshot_from_2021-03-11_23-14-57_-_1.png
Screenshot_from_2021-03-11_23-14-57_-_1.png
Attachment 2: Screenshot_from_2021-03-11_23-15-31.png
Screenshot_from_2021-03-11_23-15-31.png
Attachment 3: Screenshot_from_2021-03-11_23-16-08.png
Screenshot_from_2021-03-11_23-16-08.png
Attachment 4: Screenshot_from_2021-03-12_00-55-09.png
Screenshot_from_2021-03-12_00-55-09.png
Attachment 5: Screenshot_from_2021-03-12_00-55-35.png
Screenshot_from_2021-03-12_00-55-35.png
Attachment 6: Screenshot_from_2021-03-12_00-58-59.png
Screenshot_from_2021-03-12_00-58-59.png
Attachment 7: Screenshot_from_2021-03-12_03-35-47.png
Screenshot_from_2021-03-12_03-35-47.png
Attachment 8: Screenshot_from_2021-03-12_03-36-35.png
Screenshot_from_2021-03-12_03-36-35.png
Attachment 9: Screenshot_from_2021-03-12_03-37-20.png
Screenshot_from_2021-03-12_03-37-20.png
Attachment 10: R33_113
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260049813 ( 2219319.2 Hz)
        Other data format:    1870189 (   15960.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:      20863 (     178.0 Hz)
                               RESUME:      20864 (     178.1 Hz)
                              SYNC100:        437 (       3.7 Hz)
                              WR48-63:        437 (       3.7 Hz)
                           FEE64 disc:     442392 (    3775.5 Hz)
                             MBS info:    1385196 (   11821.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      32687 (     279.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      117.175 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1               49.525              113.280
  2               56.916              110.595
  3               47.492              115.964
  4               57.911              115.427
  5                7.968              116.501
  6       1615384952.728              108.448
  7               42.814              114.622
  8               80.186              113.011
  9               88.924               96.905
 10               21.378              111.132
 11               73.040              114.354
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22118772      49791          0          0      18934      18933         32         32      11506        354          0       4012
  1   22984048      20152          0          0        166        167         43         43      19337        396          0       5880
  2   20394651     783427          0          0        163        164         37         37     255035     527991          0       3363
  3   22242991     614086          0          0        167        167         40         40       2080     611592          0        620
  4   21439332      55343          0          0        169        170         49         49       8463      46443          0       2902
  5   16079204     115814          0          0        279        279         29         29      22786      92412          0       7566
  6   22040349     143536          0          0        183        183         31         31     102212      40896          0       2700
  7   22240015      67179          0          0        150        150         58         58       1651      65112          0        545
  8   21453279       3177          0          0        179        179         23         23       2773          0          0        850
  9   22650021       4350          0          0        185        184         20         20       3941          0          0       1026
 10   24195088      12236          0          0        107        107         38         38      11946          0          0       3047
 11   22212063       1098          0          0        181        181         37         37        662          0          0        176
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:17230.426s (    1.857 blocks/s,   0.116 Mb/s)
Attachment 11: Screenshot_from_2021-03-12_04-23-51.png
Screenshot_from_2021-03-12_04-23-51.png
Attachment 12: Screenshot_from_2021-03-12_04-29-06.png
Screenshot_from_2021-03-12_04-29-06.png
Attachment 13: Screenshot_from_2021-03-12_05-34-02.png
Screenshot_from_2021-03-12_05-34-02.png
Attachment 14: Screenshot_from_2021-03-12_05-34-28_-_1.png
Screenshot_from_2021-03-12_05-34-28_-_1.png
Attachment 15: Screenshot_from_2021-03-12_05-34-51.png
Screenshot_from_2021-03-12_05-34-51.png
  189   Thu Mar 11 07:09:25 2021 OH, LSThursday 11th of March
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
      N.B. wrong date given to the screenshots by accident (It's early)
      Statistics - attachment 1
      FEE temperatures (AIDA10 unavailable) - attachment 2
      FEE temperature - attachment 3
      Bias and leakage currents ok - attachment 4

08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
      Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
      Restarted but no stats in aida04
      It did not recover from a reset
      Telnet in and did a reboot command

09:10 Finally recovered issue was it was not linking to merger
      There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
      Manually set the ASIC settings for aida12. Rates look as expected after
      Will try to reset once beam is back
      

10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
      Analysis of R30_21 looks fine
      
10:38 System wide checks ok
      Statistics - attachment 5
      Temperature - attachment 6
      Bias and leakage - attachment 7

12.10 (LS)
      System wide checks all okay, no fails
      Statistics (attachment8)
      Rate spectra (attachment9)
      FEE temps (attament10)
      Leakage currents written to sheet (attachment11)
      Merger ~45M items/s
      Tapeserver ~14MB/s

13.45 no beam (around file R30_98)

14.00 System wide checks all okay, no fails
      Statistics (attachment12)
      FEE temps (attament13)
      Leakage currents written to sheet (attachment14)
      Merger ~44M items/s
      Tapeserver ~14MB/s
      Beam still down

14.11 beam back aida file R30_112

14.13 beam back down,  back at 14.16

14.20 no beam

14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
      double by analysing R30_101,102,103 no timewarps
      beam is back but low intensity at the moment

14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)

16.00 System wide checks all okay, no fails
      Statistics (attachment16)
      Rate spectra (attachment17)
      FEE temps (attament18)
      Leakage currents written to sheet (attachment19)
      Merger ~45M items/s
      Tapeserver ~14MB/s

18.00 System wide checks all okay, no fails
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE temps (attament22)
      Leakage currents written to sheet (attachment23)
      Merger ~45M items/s
      Tapeserver ~15MB/s

      bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
      now see error in MBS relay!
      "Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
      It triggered 63 times.
      I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser

19:10 MBS DAQ stopped to allow FRS tests

19:38 MBS DAQ is back

20:15 Aida07 fails FPGA check
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
   
      All other system wide checks ok
      Statistics - attachment 24 (Again ignore the year in the filename)
      Temperatures - attachment 25
      Bias and leakage currents - 26

21:36 They are resetting the time sorter due to FATIMA issues

22:38 System wide checks error on WR in addition to the previous FPGA error
		 Base 		Current 	Difference
aida05 fault 	 0x5cc9 : 	 0x5cca : 	 1  
aida06 fault 	 0xde76 : 	 0xde77 : 	 1  
aida07 fault 	 0xa566 : 	 0xa567 : 	 1  
aida08 fault 	 0x6ab0 : 	 0x6ab1 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4
      Statistics - attachment 27
      Temperature - attachment 28
      Bias and leakage current - attachment 29

23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain.
Attachment 1: 210310_0805_Stats.png
210310_0805_Stats.png
Attachment 2: 210310_0806_Temp.png
210310_0806_Temp.png
Attachment 3: 210310_0806_Temp2.png
210310_0806_Temp2.png
Attachment 4: 210310_0808_Bias.png
210310_0808_Bias.png
Attachment 5: 210311_1039_Stats.png
210311_1039_Stats.png
Attachment 6: 210311_1039_Temp.png
210311_1039_Temp.png
Attachment 7: 210311_1041_Bias.png
210311_1041_Bias.png
Attachment 8: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 11: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 12: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 13: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 14: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 15: 1450_SpectraRate.PNG
1450_SpectraRate.PNG
Attachment 16: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 17: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 18: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 19: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
Attachment 20: 1800_Statistics.PNG
1800_Statistics.PNG
Attachment 21: 1800_SpectraRate.PNG
1800_SpectraRate.PNG
Attachment 22: 1800_FEETemps.PNG
1800_FEETemps.PNG
Attachment 23: 1800_LeakageCurrents.PNG
1800_LeakageCurrents.PNG
Attachment 24: 200311_2014_Stats.png
200311_2014_Stats.png
Attachment 25: 200311_2014_Temp.png
200311_2014_Temp.png
Attachment 26: 210311_2017_Bias.png
210311_2017_Bias.png
Attachment 27: 210311_2225_sTATS.png
210311_2225_sTATS.png
Attachment 28: 210311_2226_Temp.png
210311_2226_Temp.png
Attachment 29: 210311_2227_Bias.png
210311_2227_Bias.png
  188   Thu Mar 11 00:38:33 2021 TDThusrday 11 March 00.00-08.00
01.38 DAQ continues OK - file R22_388

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.4M data items/s
      TapeServer OK - 14Mb/s

01.47 no merger server error/warning messages since last check by TD at start of shift 00.00

03.44 merger server error/warning messages since last check e.g.

MERGE Data Link (30260): bad timestamp  6 3 0xc1bd7e9a 0x0ce9c9c6 0x0000292e0ce9c9c6 0x166b292e0ce9c9c6 0x166b292e0cea5486

03.46 DAQ continues OK - file R22_441

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

FPGA timestamp errors
			 Base 		Current 		Difference
aida07 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.6M data items/s
      TapeServer OK - 15Mb/s

02.51 Rate spectra - attachment 7

04.14 p+n junction HEC spectra - attachment 8

      all spectra zero'd 

06.22 merger server error/warning messages since last check e.g.

MERGE Data Link (30265): bad timestamp  11 3 0xc2fc8067 0x02df391e 0x0000324e62df391e 0x166b324e62df391e 0x166b324e643c775e

06.23 DAQ continues OK - file R22_508

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status
		 Base 		Current 	Difference
aida01 fault 	 0xe67 : 	 0xe68 : 	 1  
aida02 fault 	 0x6b22 : 	 0x6b23 : 	 1  
aida03 fault 	 0x4c76 : 	 0x4c77 : 	 1  
aida04 fault 	 0x7542 : 	 0x7543 : 	 1  
aida05 fault 	 0x1597 : 	 0x1599 : 	 2  
aida06 fault 	 0xe241 : 	 0xe243 : 	 2  
aida07 fault 	 0x749 : 	 0x74c : 	 3  
aida08 fault 	 0xf854 : 	 0xf856 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors

			 Base 		Current 		Difference
aida07 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 9
      Good event statistics OK - attachment 10
      Detector bias & leakage currents OK - attachment 11
      Merger OK - 4.8M data items/s
      TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-11_Statistics_aidas-gsi.png
Screenshot_2021-03-11_Statistics_aidas-gsi.png
Attachment 3: 20.png
20.png
Attachment 4: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-11_Statistics_aidas-gsi(1).png
Screenshot_2021-03-11_Statistics_aidas-gsi(1).png
Attachment 6: 21.png
21.png
Attachment 7: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 10: Screenshot_2021-03-11_Statistics_aidas-gsi(2).png
Screenshot_2021-03-11_Statistics_aidas-gsi(2).png
Attachment 11: 22.png
22.png
  187   Wed Mar 10 07:16:02 2021 OH, LSWednesday 10th March 08:00-24:00
08:16 System wide checks all ok
      Statistics - attachment 1
      Temperatures - attachment 2
      bias - attachment 3
      Merger running 4.5e6 events per second
      Tape data writing at 42MB per second

      Currently no beam

09:45 While there is no beam we will perform a longer test of the new merger.
      Run R20 stopped and merger changed to neew version
      Started R21.

      The now usual behaviour of additional files at start of merge observed.
      N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was 
set to going
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18

     TapeData rate of 13280 kB/sec
     No errors observed in ucesb
     Correlations seen between time machine in AIDA and Ge in online
     Pulser rate in the online makes sense
     With the current data rate remaining HDD space on current drive will last around 89 hours.

09.20 analysis of file R21_27
      ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data 


10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
    (procedure "XAIDAAccessClient__Get" line 1)
    invoked from within
"XAIDAAccessClient__Get $Addr"

     aida02 restarted itself during the reset process
     Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
     System wide checks following the restart all ok.
     When restarting the MBS relay errors observed until a timestamp was observed:
     Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000

10:55 Statistics - attachment 5
      Temperature - attachment 6
      Bias - attachment 7

11:03 Time machine correlation spectra with new merger
      AIDA - FATIMA - Attachment 8
      AIDA - Ge - Attachment 9

11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the 
layout to determine which FEE/ASIC caused the events before checking ASIC control

12:12 System wide checks all ok *except adc calibration which is same as before
      Statistics - attachment 10
      Temperature - attachment 11
      Bias and leakage currents ok - attachment 12

      A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.

      After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
      They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon 
cable from the VME scalers.
      Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of 
DSSD3.
    
12:50 We have waveforms for some FEES
      Layout 7 attachment -14
      Layout 8 - attachment 15

14.00 (LS)
      Still no beam
      System wide checks okay except same as before:
      **FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module**
      
      Statistics (attachment16)
      Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
      aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
      aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
 
      FEE Temps (attachment17)
      Leakage currents written to sheets(attachment18)
      Merger ~ 45M items/s
      TapeServer~ 14MB/s
     
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file 
      R22_98, rate spectra attached (attachment19)

14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw 
      no timewarps so we are not losing anything.
      Restarted MBS relay, errors have not reappeared so far

16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE Temps (attachment22)
      Leakage currents written to sheets(attachment23)
      Merger ~ 4.3M items/s
      TapeServer~ 14MB/s  
      Current file R22_148   

16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from 
S452f115
      Around R22_150

18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
      Rates of R22_122 (attachment25):
      First DSSSD (fee0-fee3) ~66/s
      Second DSSSD (fee4-fee7) ~49/s
      Third DSSSD (fee8-fee11) ~22/s

      Rates of R22_178 (attachment26):
      First DSSSD (fee0-fee3) ~122/s
      Second DSSSD (fee4-fee7) ~94/s
      Third DSSSD (fee8-fee11) ~49/s


18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment27)
      Rate spectra (attachment28)
      FEE Temps (attachment29)
      Leakage currents written to sheets(attachment30), look to be on the way down
      Merger ~ 4.5M items/s
      TapeServer~ 15MB/s  

18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier

19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.

20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre 
reboot)
      We are borrowing a HDD from the SHIP group

20:30 System wide checks all ok
      Statistics - attachment 31
      Temperature - attachment 32
      Bias - attachment 33

22:10 MBS DAQ has crashed
      Merger terminal is now showing a large number of bad merge events

      It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
      

22:42 System wide checks all ok
      Statistics - attachment 34
      Temperature - Attachment 35
      Bias and leakage currents ok - attachment 36

23:33 System wide checks all ok - attachment 37
      Temperatures - attachment 38
      Bias and leakage currents ok - attachment 39
Attachment 1: 210310_0813_Stats.png
210310_0813_Stats.png
Attachment 2: 210310_0813_Temp.png
210310_0813_Temp.png
Attachment 3: 210310_0714_Bias.png
210310_0714_Bias.png
Attachment 4: R21_27
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259319480 ( 1687258.2 Hz)
        Other data format:    2600520 (   16920.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1272 (       8.3 Hz)
                               RESUME:       1272 (       8.3 Hz)
                              SYNC100:        573 (       3.7 Hz)
                              WR48-63:        573 (       3.7 Hz)
                           FEE64 disc:     302889 (    1970.7 Hz)
                             MBS info:    2293941 (   14925.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.693 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000              150.055
  2                0.129              153.277
  3                1.695              152.203
  4                0.000               97.174
  5               15.630              152.471
  6                0.533              122.407
  7                0.048              144.955
  8               17.221              151.934
  9               26.571              143.076
 10                0.063              148.982
 11               11.592              141.197
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20482535       1736          0          0        381        381         52         52          0        870          0          0
  1   26566394       1030          0          0          0          0         56         56          0        918          0          0
  2   26980761    1455064          0          0          2          2         87         87     302889    1151997          0          0
  3   26884821    1140284          0          0          5          5         65         65          0    1140144          0          0
  4    7516935         23          0          0          0          0          7          7          0          9          0          0
  5   16098094       1219          0          0        577        577         31         31          0          3          0          0
  6    9441595         40          0          0          2          2         18         18          0          0          0          0
  7    8769482         38          0          0          2          2         17         17          0          0          0          0
  8   32618578        336          0          0         98         98         70         70          0          0          0          0
  9   32790396        426          0          0        136        136         77         77          0          0          0          0
 10   16087963         82          0          0          1          1         40         40          0          0          0          0
 11   35081926        242          0          0         68         68         53         53          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:37059.496s (    0.863 blocks/s,   0.054 Mb/s)
Attachment 5: 210310_1054_Stats.png
210310_1054_Stats.png
Attachment 6: 210310_1055_Temp.png
210310_1055_Temp.png
Attachment 7: 210310_1055_Bias.png
210310_1055_Bias.png
Attachment 8: AIDA_Fatima_Correlation.PNG
AIDA_Fatima_Correlation.PNG
Attachment 9: AIDA_Ge_Correlation.PNG
AIDA_Ge_Correlation.PNG
Attachment 10: 210310_1210_Stats.png
210310_1210_Stats.png
Attachment 11: 210310_1211_Temp.png
210310_1211_Temp.png
Attachment 12: 210310_1212_Bias.png
210310_1212_Bias.png
Attachment 13: AIDA_leakage.PNG
AIDA_leakage.PNG
Attachment 14: 210310_1348_Layout7.png
210310_1348_Layout7.png
Attachment 15: 210310_1252_Layout8.png
210310_1252_Layout8.png
Attachment 16: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 17: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 18: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 19: 1400_SpectraRate.PNG
1400_SpectraRate.PNG
Attachment 20: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 21: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 22: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 23: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
Attachment 24: 1640_SpectraRate.PNG
1640_SpectraRate.PNG
Attachment 25: 1700_R22_122_Rates.PNG
1700_R22_122_Rates.PNG
Attachment 26: 1700_R22_178_Rates.PNG
1700_R22_178_Rates.PNG
Attachment 27: 1800_Statistics.PNG
1800_Statistics.PNG
Attachment 28: 1800_SpectraRate.PNG
1800_SpectraRate.PNG
Attachment 29: 1800_FEETemps.PNG
1800_FEETemps.PNG
Attachment 30: 1800_LeakageCurrents.PNG
1800_LeakageCurrents.PNG
Attachment 31: 210310_2039_Stats.png
210310_2039_Stats.png
Attachment 32: 210310_2040_Temp.png
210310_2040_Temp.png
Attachment 33: 210310_2041_Bias.png
210310_2041_Bias.png
Attachment 34: 210310_2241_Stats.png
210310_2241_Stats.png
Attachment 35: 210310_2241_Temp.png
210310_2241_Temp.png
Attachment 36: 210310_2242_Bias.png
210310_2242_Bias.png
Attachment 37: 210310_2331_Stats.png
210310_2331_Stats.png
Attachment 38: 210310_2332_Temp.png
210310_2332_Temp.png
Attachment 39: 210310_2333_Bias.png
210310_2333_Bias.png
  186   Tue Mar 9 23:12:17 2021 CAMarch 10th 00:00 - 08:00
00:13 beam back, but some spills 'missing'

      DESPEC keeping beam as is, but may stop later if it worsens

00:15 DAQ continues OK - file R20_622

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:20 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

     If any of these counts are reported as in error
     The ASIC readout system has detected a timeslip.
     That is the timestamp read from the time FIFO is not younger than the last

00:24 still no bad timestamp errors in NewMerger since 16:30 UTC

00:30 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3

      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

01:30 rate spectra - attachment 4

      HEC spectra - attachment 5 & 6

      note aida04 spectrum still not showing

02:07 beam off - file R20_760

02:11 beam back - file R20_764

02:44 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      still no bad timestamp errors in NewMerger since 16:30 UTC

      FEE64 Temperatures OK - attachment 7
      Good event statistics OK - attachment 8
      Detector bias & leakage currents OK - attachment 9

      Merger OK - 4.2M data items/s
      TapeServer OK - 44 Mb/s

04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10

      System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      FEE64 Temperatures OK - attachment 11
      Good event statistics OK - attachment 12
      Detector bias & leakage currents OK - attachment 13

05:03 Merger OK - 4.5M data items/s
      TapeServer OK - 43 Mb/s

      no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate

06:11 rates spectra - attachment 14

06:14 System wide checks

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      Check WR decoder status
      Base 		Current 	Difference
      aida05 fault 	 0x1591 : 	 0x1593 : 	 2  
      aida06 fault 	 0xe23c : 	 0xe23e : 	 2  
      aida07 fault 	 0x743 : 	 0x745 : 	 2  
      aida08 fault 	 0xf84e : 	 0xf850 : 	 2  
      aida09 fault 	 0x5449 : 	 0x544a : 	 1  
      White Rabbit error counter test result: Passed 7, Failed 5

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
   

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

      still no further bad timestamp errors in NewMerger terminal

      collected all WR and FPGA errors from baseline, system wide checks ok

      FEE64 Temperatures OK - attachment 15
      Good event statistics OK - attachment 16
      Detector bias & leakage currents OK - attachment 17

06:24 AIDA writing to file R20_1080

      Merger OK - 4.5M data items/s
      TapeServer OK - 46 Mb/s

06:26 another burst of bad timestamp errors - attachment 18

      performed system wide checks again - all ok aside from aida07/aida10 calibration

07:32 HEC spectra - attachments 19/20

07:35 no beam

      


      
Attachment 1: Screenshot_from_2021-03-09_23-27-20.png
Screenshot_from_2021-03-09_23-27-20.png
Attachment 2: Screenshot_from_2021-03-09_23-29-22.png
Screenshot_from_2021-03-09_23-29-22.png
Attachment 3: Screenshot_from_2021-03-09_23-29-39.png
Screenshot_from_2021-03-09_23-29-39.png
Attachment 4: Screenshot_from_2021-03-10_00-27-45.png
Screenshot_from_2021-03-10_00-27-45.png
Attachment 5: Screenshot_from_2021-03-10_00-28-39.png
Screenshot_from_2021-03-10_00-28-39.png
Attachment 6: Screenshot_from_2021-03-10_00-29-25.png
Screenshot_from_2021-03-10_00-29-25.png
Attachment 7: Screenshot_from_2021-03-10_01-40-24.png
Screenshot_from_2021-03-10_01-40-24.png
Attachment 8: Screenshot_from_2021-03-10_01-42-57.png
Screenshot_from_2021-03-10_01-42-57.png
Attachment 9: Screenshot_from_2021-03-10_01-43-39_-_1.png
Screenshot_from_2021-03-10_01-43-39_-_1.png
Attachment 10: Screenshot_from_2021-03-10_03-17-56.png
Screenshot_from_2021-03-10_03-17-56.png
Attachment 11: Screenshot_from_2021-03-10_03-23-03.png
Screenshot_from_2021-03-10_03-23-03.png
Attachment 12: Screenshot_from_2021-03-10_03-23-26.png
Screenshot_from_2021-03-10_03-23-26.png
Attachment 13: Screenshot_from_2021-03-10_03-23-43.png
Screenshot_from_2021-03-10_03-23-43.png
Attachment 14: Screenshot_from_2021-03-10_05-10-19.png
Screenshot_from_2021-03-10_05-10-19.png
Attachment 15: Screenshot_from_2021-03-10_05-17-39.png
Screenshot_from_2021-03-10_05-17-39.png
Attachment 16: Screenshot_from_2021-03-10_05-18-04.png
Screenshot_from_2021-03-10_05-18-04.png
Attachment 17: Screenshot_from_2021-03-10_05-21-47.png
Screenshot_from_2021-03-10_05-21-47.png
Attachment 18: Screenshot_from_2021-03-10_05-27-03.png
Screenshot_from_2021-03-10_05-27-03.png
Attachment 19: Screenshot_from_2021-03-10_06-31-51.png
Screenshot_from_2021-03-10_06-31-51.png
Attachment 20: Screenshot_from_2021-03-10_06-32-18.png
Screenshot_from_2021-03-10_06-32-18.png
  185   Tue Mar 9 20:04:20 2021 TDTuesday 9 March
21.04 DAQ continues OK - file R20_379

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      System wide checks all OK *except*

ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

Check FPGA Timestamp Errors
Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1

If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

21.20 no merger server error/warning messages since 16.30UTC

21.35 p+n junction strip  HEC & LEC spectra, Rate spectra common y-scale 0-30000 & 0-10
      all spectra zero'd

22.05 no beam for 2-3h whilst accelerator generator is fixed
Attachment 1: Screenshot_2021-03-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-09_Statistics_aidas-gsi.png
Screenshot_2021-03-09_Statistics_aidas-gsi.png
Attachment 3: 10.png
10.png
Attachment 4: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(3).png
  184   Tue Mar 9 10:23:45 2021 TDAnalysis of R14 (w/ NewMerger with min info code 4 & 5 data items)
Per https://elog.ph.ed.ac.uk/DESPEC/183 R14 data files have been analysed

I think we still need  paired info code 4 & 5 data items.
I also don't understand the pattern of file sizes ... 'old' TapeServer buffers being flushed?

[npg@aidas-gsi analyser]$ ls -lrth /TapeData/S452/R14*
-rw-rw-r--. 1 npg npg  64K Mar  9 10:07 /TapeData/S452/R14_0
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_1
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_2
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_3
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_4
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_5
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_6
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_7
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_8
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_9
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_10
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_11
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_12
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_13
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_14
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_15
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_16
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_17
-rw-rw-r--. 1 npg npg 363M Mar  9 10:23 /TapeData/S452/R14_18
-rw-rw-r--. 1 npg npg 2.0G Mar  9 10:26 /TapeData/S452/R14_19
-rw-rw-r--. 1 npg npg 1.6G Mar  9 10:28 /TapeData/S452/R14_20

R14_0 (attachment 1) has multiple PAUSE/RESUME data items at start of file (R14_0) followed by 1x info code 4 & 5 data items (as expected)

Also true for other short (64k) files - multiple PAUSE/RESUME data items, 1x info code 4 & 5 data items, and ADC data from one FEE64 module only

[npg@aidas-gsi analyser]$ ./analyser v /TapeData/S452/R14_0 | more
 *** TDR format 3.3.0 analyser - TD - January 2019
  verbose
 ***    RESUME timestamp: block:     1 ptr:     7 data: 0x843A160B module:  4 information type:  3 information field: 0x000A160B ts: 0x0000A160B1733D66
 ***     PAUSE timestamp: block:     1 ptr:     9 data: 0x842A160B module:  4 information type:  2 information field: 0x000A160B ts: 0x0000A160B6508028
 ***    RESUME timestamp: block:     1 ptr:    11 data: 0x873A160B module:  7 information type:  3 information field: 0x000A160B ts: 0x0000A160B7727448
 ***    RESUME timestamp: block:     1 ptr:    13 data: 0x823A160B module:  2 information type:  3 information field: 0x000A160B ts: 0x0000A160B775CA9E
 ***     PAUSE timestamp: block:     1 ptr:    15 data: 0x872A160B module:  7 information type:  2 information field: 0x000A160B ts: 0x0000A160BC4F8028
 ***     PAUSE timestamp: block:     1 ptr:    17 data: 0x822A160B module:  2 information type:  2 information field: 0x000A160B ts: 0x0000A160BC548028
 ***    RESUME timestamp: block:     1 ptr:    19 data: 0x863A160B module:  6 information type:  3 information field: 0x000A160B ts: 0x0000A160BFE804B2
 ***     PAUSE timestamp: block:     1 ptr:    21 data: 0x862A160C module:  6 information type:  2 information field: 0x000A160C ts: 0x0000A160C4C70028
 ***    RESUME timestamp: block:     1 ptr:    23 data: 0x803A160C module:  0 information type:  3 information field: 0x000A160C ts: 0x0000A160C863646A
 ***    RESUME timestamp: block:     1 ptr:    25 data: 0x833A160C module:  3 information type:  3 information field: 0x000A160C ts: 0x0000A160CC641992
 ***     PAUSE timestamp: block:     1 ptr:    27 data: 0x802A160C module:  0 information type:  2 information field: 0x000A160C ts: 0x0000A160CD410028
 ***     PAUSE timestamp: block:     1 ptr:    29 data: 0x832A160D module:  3 information type:  2 information field: 0x000A160D ts: 0x0000A160D1420028
 ***    RESUME timestamp: block:     1 ptr:    31 data: 0x853A160D module:  5 information type:  3 information field: 0x000A160D ts: 0x0000A160D3D39E8C
 ***    RESUME timestamp: block:     1 ptr:    33 data: 0x883A160D module:  8 information type:  3 information field: 0x000A160D ts: 0x0000A160D3D4D5EA
 ***     PAUSE timestamp: block:     1 ptr:    35 data: 0x852A160D module:  5 information type:  2 information field: 0x000A160D ts: 0x0000A160D8B30028
 ***     PAUSE timestamp: block:     1 ptr:    37 data: 0x882A160D module:  8 information type:  2 information field: 0x000A160D ts: 0x0000A160D8B30028
 ***    RESUME timestamp: block:     1 ptr:    39 data: 0x813A160D module:  1 information type:  3 information field: 0x000A160D ts: 0x0000A160D9F76CF8
 ***     PAUSE timestamp: block:     1 ptr:    41 data: 0x812A160D module:  1 information type:  2 information field: 0x000A160D ts: 0x0000A160DED50028
 ***    RESUME timestamp: block:     1 ptr:    43 data: 0x843A160F module:  4 information type:  3 information field: 0x000A160F ts: 0x0000A160F7AD970E
 ***    RESUME timestamp: block:     1 ptr:    45 data: 0x8A3A160F module: 10 information type:  3 information field: 0x000A160F ts: 0x0000A160F938391C
 ***    RESUME timestamp: block:     1 ptr:    47 data: 0x893A160F module:  9 information type:  3 information field: 0x000A160F ts: 0x0000A160FDC0600E
 ***     PAUSE timestamp: block:     1 ptr:    49 data: 0x8A2A160F module: 10 information type:  2 information field: 0x000A160F ts: 0x0000A160FE178028
 ***    RESUME timestamp: block:     1 ptr:    51 data: 0x823A1610 module:  2 information type:  3 information field: 0x000A1610 ts: 0x0000A161009DF9B2
 ***    RESUME timestamp: block:     1 ptr:    53 data: 0x873A1610 module:  7 information type:  3 information field: 0x000A1610 ts: 0x0000A16100B7FD08
 ***     PAUSE timestamp: block:     1 ptr:    55 data: 0x892A1610 module:  9 information type:  2 information field: 0x000A1610 ts: 0x0000A161029F8028
 ***    RESUME timestamp: block:     1 ptr:    57 data: 0x863A1610 module:  6 information type:  3 information field: 0x000A1610 ts: 0x0000A16108134396
 ***    RESUME timestamp: block:     1 ptr:    59 data: 0x803A1610 module:  0 information type:  3 information field: 0x000A1610 ts: 0x0000A1610C484524
 ***    RESUME timestamp: block:     1 ptr:    61 data: 0x833A1610 module:  3 information type:  3 information field: 0x000A1610 ts: 0x0000A1610F8B0910
 ***    RESUME timestamp: block:     1 ptr:    63 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16112BA0000
 ***    RESUME timestamp: block:     1 ptr:    65 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16112E48000
 ***    RESUME timestamp: block:     1 ptr:    67 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161130F0000
 ***    RESUME timestamp: block:     1 ptr:    69 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113398000
 ***    RESUME timestamp: block:     1 ptr:    71 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113640000
 ***    RESUME timestamp: block:     1 ptr:    73 data: 0x883A1611 module:  8 information type:  3 information field: 0x000A1611 ts: 0x0000A161137B1560
 ***    RESUME timestamp: block:     1 ptr:    75 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161138E8000
 ***    RESUME timestamp: block:     1 ptr:    77 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113B90000
 ***    RESUME timestamp: block:     1 ptr:    79 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113E38000
 ***    RESUME timestamp: block:     1 ptr:    81 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161140E0000
 ***    RESUME timestamp: block:     1 ptr:    83 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114388000
 ***    RESUME timestamp: block:     1 ptr:    85 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114630000
 ***    RESUME timestamp: block:     1 ptr:    87 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161148D8000
 ***    RESUME timestamp: block:     1 ptr:    89 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114B80000
 ***    RESUME timestamp: block:     1 ptr:    91 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114E28000
 ***    RESUME timestamp: block:     1 ptr:    93 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161150D0000
 ***    RESUME timestamp: block:     1 ptr:    95 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115378000
 ***    RESUME timestamp: block:     1 ptr:    97 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115620000
 ***    RESUME timestamp: block:     1 ptr:    99 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161158C8000
 ***    RESUME timestamp: block:     1 ptr:   101 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115B70000
 ***    RESUME timestamp: block:     1 ptr:   103 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115E18000
 ***    RESUME timestamp: block:     1 ptr:   105 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161160C0000
 ***    RESUME timestamp: block:     1 ptr:   107 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116368000
 ***    RESUME timestamp: block:     1 ptr:   109 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116610000
 ***    RESUME timestamp: block:     1 ptr:   111 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161168B8000
 ***    RESUME timestamp: block:     1 ptr:   113 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116B60000
 ***    RESUME timestamp: block:     1 ptr:   115 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116E08000
 ***    RESUME timestamp: block:     1 ptr:   117 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161170B0000
 ***    RESUME timestamp: block:     1 ptr:   119 data: 0x853A1611 module:  5 information type:  3 information field: 0x000A1611 ts: 0x0000A161171BADC4
 ***    RESUME timestamp: block:     1 ptr:   121 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16117358000
 ***    RESUME timestamp: block:     1 ptr:   123 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16117600000
 ***    RESUME timestamp: block:     1 ptr:   125 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161178A8000
 ***    RESUME timestamp: block:     1 ptr:   127 data: 0x813A1611 module:  1 information type:  3 information field: 0x000A1611 ts: 0x0000A1611A161F3C
 ***    RESUME timestamp: block:     1 ptr:   129 data: 0x8A3A1613 module: 10 information type:  3 information field: 0x000A1613 ts: 0x0000A1613C6AAE72
 ***    RESUME timestamp: block:     1 ptr:   131 data: 0x893A1613 module:  9 information type:  3 information field: 0x000A1613 ts: 0x0000A1613F230BBE
 ***   WR48-64 timestamp: block:     1 ptr:   133 data: 0x8450166A module:  4 information type:  5 information field: 0x0000166A ts: 0x166AA1613483FE84
 ***   SYNC100 timestamp: block:     1 ptr:   135 data: 0x844A1622 module:  4 information type:  4 information field: 0x000A1622 ts: 0x166AA1622483FE84
 ***            ADC data: block:     1 ptr:   137 data: 0xC13C7FEB module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32747  ts: 0x166AA1622483FE84
 ***            ADC data: block:     1 ptr:   139 data: 0xC1067E53 module:  4 fail: 0 range: 0 id:   262 channel:  6 adc: 32339  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   141 data: 0xC1167E4C module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32332  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   143 data: 0xC1247EAF module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32431  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   145 data: 0xC11F7D3F module:  4 fail: 0 range: 0 id:   287 channel: 31 adc: 32063  ts: 0x166AA16224840E24
 ***            ADC data: block:     1 ptr:   147 data: 0xC12D7EF0 module:  4 fail: 0 range: 0 id:   301 channel: 45 adc: 32496  ts: 0x166AA16224840E24
 ***            ADC data: block:     1 ptr:   149 data: 0xC1117E24 module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32292  ts: 0x166AA16224842594
 ***            ADC data: block:     1 ptr:   151 data: 0xC11E7D06 module:  4 fail: 0 range: 0 id:   286 channel: 30 adc: 32006  ts: 0x166AA16224844CA4
 ***            ADC data: block:     1 ptr:   153 data: 0xC1217EB3 module:  4 fail: 0 range: 0 id:   289 channel: 33 adc: 32435  ts: 0x166AA16224845474
 ***            ADC data: block:     1 ptr:   155 data: 0xC1347E05 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32261  ts: 0x166AA16224845474
 ***            ADC data: block:     1 ptr:   157 data: 0xC1037EBC module:  4 fail: 0 range: 0 id:   259 channel:  3 adc: 32444  ts: 0x166AA16224845C44
 ***            ADC data: block:     1 ptr:   159 data: 0xC13E7E99 module:  4 fail: 0 range: 0 id:   318 channel: 62 adc: 32409  ts: 0x166AA16224845C44
 ***            ADC data: block:     1 ptr:   161 data: 0xC1047D6F module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32111  ts: 0x166AA16224846414
 ***            ADC data: block:     1 ptr:   163 data: 0xC1137EE5 module:  4 fail: 0 range: 0 id:   275 channel: 19 adc: 32485  ts: 0x166AA16224846414
 ***            ADC data: block:     1 ptr:   165 data: 0xC10E7F30 module:  4 fail: 0 range: 0 id:   270 channel: 14 adc: 32560  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   167 data: 0xC1157D12 module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32018  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   169 data: 0xC1287DD3 module:  4 fail: 0 range: 0 id:   296 channel: 40 adc: 32211  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   171 data: 0xC1197E69 module:  4 fail: 0 range: 0 id:   281 channel: 25 adc: 32361  ts: 0x166AA162248473B4
 ***            ADC data: block:     1 ptr:   173 data: 0xC11B7EC0 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32448  ts: 0x166AA16224847B84
 ***            ADC data: block:     1 ptr:   175 data: 0xC1017D82 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32130  ts: 0x166AA16224848354
 ***            ADC data: block:     1 ptr:   177 data: 0xC1247E94 module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32404  ts: 0x166AA162248492F4
 ***            ADC data: block:     1 ptr:   179 data: 0xC1047D33 module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32051  ts: 0x166AA1622484F884
 ***            ADC data: block:     1 ptr:   181 data: 0xC13C7FFC module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32764  ts: 0x166AA16224853ED4
 ***            ADC data: block:     1 ptr:   183 data: 0xC1067E6A module:  4 fail: 0 range: 0 id:   262 channel:  6 adc: 32362  ts: 0x166AA16224855E14
 ***            ADC data: block:     1 ptr:   185 data: 0xC1117E44 module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32324  ts: 0x166AA16224858CF4
 ***            ADC data: block:     1 ptr:   187 data: 0xC1167EA2 module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32418  ts: 0x166AA162248594C4
 ***            ADC data: block:     1 ptr:   189 data: 0xC1017EFB module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32507  ts: 0x166AA1622485B404
 ***            ADC data: block:     1 ptr:   191 data: 0xC1017F83 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32643  ts: 0x166AA16224864874
 ***            ADC data: block:     1 ptr:   193 data: 0xC1197E3C module:  4 fail: 0 range: 0 id:   281 channel: 25 adc: 32316  ts: 0x166AA162248667B4
 ***            ADC data: block:     1 ptr:   195 data: 0xC1017DF9 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32249  ts: 0x166AA1622486DCE4
 ***            ADC data: block:     1 ptr:   197 data: 0xC1117E4E module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32334  ts: 0x166AA162248759E4
 ***            ADC data: block:     1 ptr:   199 data: 0xC1167E55 module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32341  ts: 0x166AA1622487AFD4
 ***            ADC data: block:     1 ptr:   201 data: 0xC1247E58 module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32344  ts: 0x166AA1622487B7A4
 ***            ADC data: block:     1 ptr:   203 data: 0xC10B7DFE module:  4 fail: 0 range: 0 id:   267 channel: 11 adc: 32254  ts: 0x166AA1622487CF14
 ***            ADC data: block:     1 ptr:   205 data: 0xC12B7E6F module:  4 fail: 0 range: 0 id:   299 channel: 43 adc: 32367  ts: 0x166AA1622487CF14
 ***            ADC data: block:     1 ptr:   207 data: 0xC1157D3B module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32059  ts: 0x166AA1622487D6E4
 ***            ADC data: block:     1 ptr:   209 data: 0xC1347E96 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32406  ts: 0x166AA1622487D6E4
 ***            ADC data: block:     1 ptr:   211 data: 0xC11A7D38 module:  4 fail: 0 range: 0 id:   282 channel: 26 adc: 32056  ts: 0x166AA1622487DEB4
 ***            ADC data: block:     1 ptr:   213 data: 0xC13C7F8E module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32654  ts: 0x166AA1622487DEB4
 ***            ADC data: block:     1 ptr:   215 data: 0xC11B7E79 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32377  ts: 0x166AA1622487E684
 ***            ADC data: block:     1 ptr:   217 data: 0xC13D7EDB module:  4 fail: 0 range: 0 id:   317 channel: 61 adc: 32475  ts: 0x166AA1622487E684
 ***            ADC data: block:     1 ptr:   219 data: 0xC1347EB8 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32440  ts: 0x166AA16224886B54
 ***            ADC data: block:     1 ptr:   221 data: 0xC13C7F67 module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32615  ts: 0x166AA16224887324
 ***            ADC data: block:     1 ptr:   223 data: 0xC13D7F18 module:  4 fail: 0 range: 0 id:   317 channel: 61 adc: 32536  ts: 0x166AA1622488F7F4
 ***            ADC data: block:     1 ptr:   225 data: 0xC1157D5C module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32092  ts: 0x166AA16224890794
 ***            ADC data: block:     1 ptr:   227 data: 0xC1047DED module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32237  ts: 0x166AA16224891F04
 ***            ADC data: block:     1 ptr:   229 data: 0xC1127E66 module:  4 fail: 0 range: 0 id:   274 channel: 18 adc: 32358  ts: 0x166AA16224891F04
--More--

R14_19 (attachment 2) looks OK - ignore rate information as this is based upon incomplete timestamp data
Attachment 1: R14_0
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:          1
          ADC data format:       8120 (   96053.8 Hz)
        Other data format:         65 (     768.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         11 (     130.1 Hz)
                               RESUME:         52 (     615.1 Hz)
                              SYNC100:          1 (      11.8 Hz)
                              WR48-63:          1 (      11.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:        0.085 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.994                0.000
  2                1.146                0.000
  3                1.045                0.000
  4                1.097                0.000
  5                1.047                0.000
  6                1.129                0.000
  7                1.148                0.000
  8                0.986                0.000
  9                1.015                0.000
 10                1.046                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0          0          3          0          0          1          2          0          0          0          0          0          0
  1          0          3          0          0          1          2          0          0          0          0          0          0
  2          0          3          0          0          1          2          0          0          0          0          0          0
  3          0          3          0          0          1          2          0          0          0          0          0          0
  4       8120          5          0          0          1          2          1          1          0          0          0          0
  5          0          3          0          0          1          2          0          0          0          0          0          0
  6          0          3          0          0          1          2          0          0          0          0          0          0
  7          0          3          0          0          1          2          0          0          0          0          0          0
  8          0          3          0          0          1          2          0          0          0          0          0          0
  9          0          3          0          0          1          2          0          0          0          0          0          0
 10          0          3          0          0          1          2          0          0          0          0          0          0
 11          0         30          0          0          0         30          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    0.004s (  256.000 blocks/s,  16.000 Mb/s)
Attachment 2: R14_19
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261598852 ( 1789659.8 Hz)
        Other data format:     321148 (    2197.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1009 (       6.9 Hz)
                               RESUME:       1010 (       6.9 Hz)
                              SYNC100:        545 (       3.7 Hz)
                              WR48-63:        545 (       3.7 Hz)
                           FEE64 disc:     317229 (    2170.2 Hz)
                             MBS info:        810 (       5.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      146.172 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.992              139.586
  2                0.853              142.002
  3                1.182              142.271
  4                0.575              139.050
  5                6.293              131.265
  6                0.208              139.586
  7                0.037              142.002
  8               39.955              138.244
  9                9.898              143.613
 10                0.448              141.465
 11               32.263              145.760
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21196377       1535          0          0        326        327         36         36          0        810          0          0
  1   25527261        100          0          0          4          4         46         46          0          0          0          0
  2   26242581     317335          0          0          6          6         47         47     317229          0          0          0
  3   27512430        132          0          0          5          5         61         61          0          0          0          0
  4    8193786         46          0          0          4          4         19         19          0          0          0          0
  5   17476404        516          0          0        220        220         38         38          0          0          0          0
  6   12155075         60          0          0          5          5         25         25          0          0          0          0
  7    9702993         52          0          0          1          1         25         25          0          0          0          0
  8   33325064        534          0          0        206        206         61         61          0          0          0          0
  9   34533675        282          0          0         56         56         85         85          0          0          0          0
 10   12187954         94          0          0          2          2         45         45          0          0          0          0
 11   33545252        462          0          0        174        174         57         57          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:40890.051s (    0.783 blocks/s,   0.049 Mb/s)
  183   Tue Mar 9 06:58:14 2021 CA, LSMarch 9th 08:00-11:00 shift
08:00 DAQ continues ok - writing to file R13_120

      Merger ok - 4.6M data items/s

      TapeServer ok - 43 MB/s

08:03 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

08:08 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08:32 CA briefly lost network connection - back now

08:42 DESPEC on file 102

09:24 Rate spectra - attachment 4

09:32 HEC spectra - attachment 5/6

      beam off - writing to file R13_243

09:42 merger bad timestamp errors - attachment 7

      all system wide checks ok

10:30 NH - Beam was down, I try the new merger from Vic again (R14)
           I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef
           After fixing my mistakes, all running with new merger and rate 15 MB/s (R14)
           It all looked OK even going to MBS/Go4
           I reverted back to original hot merger, R15 and look at the data.
           Beam is now back and I return AIDA to shifters :)

11.30  System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Reran calibration for both FEEs  but still fail
       Statistics (attachment8)
       Rate spectra (attachment9)
       FEE temps (attachment10)
       Leakage currents written to sheet (attachment11)
       Tapeserver ~33MB/s
       Merger ~ 3.4M items/s
       AIDA file R15_92

11.55 Beam off for a couple minutes but back now

12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working
      Now writing to R16
      Merger ~ 4.5M items/s
      Tapeserver ~44MB/s

12.41 no beam

13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down
      Beam is back now, forwarding to MBS
      Merger ~ 45M items/s
      Tapeserver ~44MB/s
      Writing to R17

13.20 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment12)
       Rate spectra (attachment13)
       FEE temps (attachment14)
       Leakage currents written to sheet (attachment15)
       Tapeserver ~45MB/s
       Merger ~ 4.4M items/s

14.27 beam off
      file R17_115
      back a couple minutes later

14.40 no beam again back at 14.46

15.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment16)
       Rate spectra (attachment17)
       FEE temps (attachment18)
       Leakage currents written to sheet (attachment19)
       Tapeserver ~44MB/s
       Merger ~ 4.4M items/s
       
15.52 beam down R17_221, back at 15.58 R17_229

16.07 Beam down
       
16:20 Removed S480 files from /media/1e... and /media/SecondDrive
      Set merger to no output and stopped tapedata.
      Changed symbolic link of /TapeData to /media/1e.....
      Restarted tapedata and the merger with outputting to file R20
      Local DESPEC was made aware of this before carrying out and stopped the run.
      Beam back

17.12 beam down for a couple of minutes R20_90

17.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment20)
       Rate spectra (attachment21)
       FEE temps (attachment22)
       Leakage currents written to sheet (attachment23)
       Tapeserver ~45MB/s
       Merger ~ 4.5M items/s

      
Attachment 1: Screenshot_from_2021-03-09_07-05-35.png
Screenshot_from_2021-03-09_07-05-35.png
Attachment 2: Screenshot_from_2021-03-09_07-06-17.png
Screenshot_from_2021-03-09_07-06-17.png
Attachment 3: Screenshot_from_2021-03-09_07-06-53.png
Screenshot_from_2021-03-09_07-06-53.png
Attachment 4: Screenshot_from_2021-03-09_08-08-45_-_1.png
Screenshot_from_2021-03-09_08-08-45_-_1.png
Attachment 5: Screenshot_from_2021-03-09_08-24-06.png
Screenshot_from_2021-03-09_08-24-06.png
Attachment 6: Screenshot_from_2021-03-09_08-26-47.png
Screenshot_from_2021-03-09_08-26-47.png
Attachment 7: Screenshot_from_2021-03-09_08-44-17.png
Screenshot_from_2021-03-09_08-44-17.png
Attachment 8: 1130_Statistics.PNG
1130_Statistics.PNG
Attachment 9: 1130_SpectraRates.PNG
1130_SpectraRates.PNG
Attachment 10: 1130_FEETemps.PNG
1130_FEETemps.PNG
Attachment 11: 1130_LeakageCurrents.PNG
1130_LeakageCurrents.PNG
Attachment 12: 1330_Statistics.PNG
1330_Statistics.PNG
Attachment 13: 1330_SpectraRates.PNG
1330_SpectraRates.PNG
Attachment 14: 1330_FEETemps.PNG
1330_FEETemps.PNG
Attachment 15: 1330_LeakageCurrents.PNG
1330_LeakageCurrents.PNG
Attachment 16: 1540_Statistics.PNG
1540_Statistics.PNG
Attachment 17: 1540_SpectraRates.PNG
1540_SpectraRates.PNG
Attachment 18: 1540_FEETemps.PNG
1540_FEETemps.PNG
Attachment 19: 1540_LeakageCurrents.PNG
1540_LeakageCurrents.PNG
Attachment 20: 1730_Statistics.PNG
1730_Statistics.PNG
Attachment 21: 1730_SpectraRates.PNG
1730_SpectraRates.PNG
Attachment 22: 1730_FEETemps.PNG
1730_FEETemps.PNG
Attachment 23: 1730_LeakageCurrents.PNG
1730_LeakageCurrents.PNG
  182   Mon Mar 8 22:51:47 2021 OHTuesday 9th March
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)

at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
          Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
          Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message

01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event

Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.


02:00 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

03:10 Problem with accelerator will be no beam for ~30 minutes


03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
      There are now also errors in all FEEs for WR events
	
		 Base 		Current 	Difference
aida01 fault 	 0xc45f : 	 0xc46a : 	 11  
aida02 fault 	 0xd250 : 	 0xd25b : 	 11  
aida03 fault 	 0x27b2 : 	 0x27bd : 	 11  
aida04 fault 	 0x714b : 	 0x7156 : 	 11  
aida05 fault 	 0x74f5 : 	 0x7500 : 	 11  
aida06 fault 	 0x558a : 	 0x5595 : 	 11  
aida07 fault 	 0x8d20 : 	 0x8d2b : 	 11  
aida08 fault 	 0x8131 : 	 0x813c : 	 11  
aida09 fault 	 0xf412 : 	 0xf41d : 	 11  
aida10 fault 	 0x82c2 : 	 0x82cd : 	 11  
aida11 fault 	 0x2845 : 	 0x2850 : 	 11  
aida12 fault 	 0x1bf6 : 	 0x1c01 : 	 11  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Also errors in FPGA
			 Base 		Current 		Difference
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x1 : 	 1  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A8D22 3A5FDE66 , WR/10=>   23DDAE9D2A32FD7, Readout Time =>   23DDAE9D3C9C000 
aida02 : White Rabbit=>  166A8D22 4C6C3AEB , WR/10=>   23DDAE9D4713917, Readout Time =>   23DDAE9D5B94000 
aida03 : White Rabbit=>  166A8D22 5FB33A3C , WR/10=>   23DDAE9D65EB906, Readout Time =>   23DDAE9D769C000 
aida04 : White Rabbit=>  166A8D22 6FBA29DB , WR/10=>   23DDAE9D7F9042F, Readout Time =>   23DDAE9D8D14000 
aida05 : White Rabbit=>  166A8D22 7F71248D , WR/10=>   23DDAE9D98B5074, Readout Time =>   23DDAE9DA9D8000 
aida06 : White Rabbit=>  166A8D22 8F78B071 , WR/10=>   23DDAE9DB25AB3E, Readout Time =>   23DDAE9DC280000 
aida07 : White Rabbit=>  166A8D22 9F5A64B7 , WR/10=>   23DDAE9DCBC3D45, Readout Time =>   23DDAE9DDFF0000 
aida08 : White Rabbit=>  166A8D22 B1E91092 , WR/10=>   23DDAE9DE974E75, Readout Time =>   23DDAE9DFA08000 
aida09 : White Rabbit=>  166A8D22 C150D9ED , WR/10=>   23DDAE9E021AF64, Readout Time =>   23DDAE9E0FD8000 
aida10 : White Rabbit=>  166A8D22 D02BA583 , WR/10=>   23DDAE9E19DF6F3, Readout Time =>   23DDAE9E2D50000 
aida11 : White Rabbit=>  166A8D22 E287CFC2 , WR/10=>   23DDAE9E373FB2D, Readout Time =>   23DDAE9E4720000 
aida12 : White Rabbit=>  166A8D22 F2D4C911 , WR/10=>   23DDAE9E515474E, Readout Time =>   23DDAE9E1BCC000 

System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results

AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).

03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.

All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.

05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
      The problem was with the accelerator operators system. There was no beam.

06:20 Error in WR system wide checks. Rest all ok
      This time there is no bad merge items recorded in the merger terminal.
      There are also no timewarp events seen in ucesb which were observed with the issues at 3am.

		 Base 		Current 	Difference
aida01 fault 	 0xdbe9 : 	 0xdbea : 	 1  
aida02 fault 	 0xa93d : 	 0xa93e : 	 1  
aida03 fault 	 0x8fc4 : 	 0x8fc5 : 	 1  
aida04 fault 	 0x3e56 : 	 0x3e57 : 	 1  
aida05 fault 	 0xb04c : 	 0xb04d : 	 1  
aida06 fault 	 0x5576 : 	 0x5577 : 	 1  
aida07 fault 	 0x99ff : 	 0x9a00 : 	 1  
aida08 fault 	 0x8412 : 	 0x8413 : 	 1  
aida09 fault 	 0x2457 : 	 0x2458 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A9523 11478D1D , WR/10=>   23DDBB6B4ED8E1C, Readout Time =>   23DDBB6B6148000 
aida02 : White Rabbit=>  166A9523 236AA5E8 , WR/10=>   23DDBB6B6BDDD64, Readout Time =>   23DDBB6B8164000 
aida03 : White Rabbit=>  166A9523 375F235D , WR/10=>   23DDBB6B8BCB6BC, Readout Time =>   23DDBB6B9C3C000 
aida04 : White Rabbit=>  166A9523 47B8038C , WR/10=>   23DDBB6BA5F338E, Readout Time =>   23DDBB6BA77C000 
aida05 : White Rabbit=>  166A9523 599FC116 , WR/10=>   23DDBB6BC29934F, Readout Time =>   23DDBB6BD34C000 
aida06 : White Rabbit=>  166A9523 696EB0A7 , WR/10=>   23DDBB6BDBE44DD, Readout Time =>   23DDBB6BECB8000 
aida07 : White Rabbit=>  166A9523 7982236B , WR/10=>   23DDBB6BF59D057, Readout Time =>   23DDBB6C0418000 
aida08 : White Rabbit=>  166A9523 87C85008 , WR/10=>   23DDBB6C0C73B34, Readout Time =>   23DDBB6C1C7C000 
aida09 : White Rabbit=>  166A9523 97020497 , WR/10=>   23DDBB6C24D0075, Readout Time =>   23DDBB6C34B4000 
aida10 : White Rabbit=>  166A9523 A69B171D , WR/10=>   23DDBB6C3DC4F1C, Readout Time =>   23DDBB6C50A0000 
aida11 : White Rabbit=>  166A9523 B86C8D7D , WR/10=>   23DDBB6C5A4748C, Readout Time =>   23DDBB6C6C88000 
aida12 : White Rabbit=>  166A9523 CA1DD1B2 , WR/10=>   23DDBB6C76961C5, Readout Time =>   23DDBB6C5C40000 

Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s) 
Cannot check file as was not writing to file.

      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10

06:40 There are occasional seconds of beam. According to operators it comes and then it goes again

06:45 Beam fairly stable so we are running again.
      AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19

07:00 Beam stopped again
07:02 Beam returned
      There are apparently issues with MUSIC chamber one and its resolution

07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. 
Attachment 1: 210308_2354_Stats.png
210308_2354_Stats.png
Attachment 2: 210308_2355_Temp.png
210308_2355_Temp.png
Attachment 3: 210308_2357_Bias.png
210308_2357_Bias.png
Attachment 4: 210309_0104_Stats.png
210309_0104_Stats.png
Attachment 5: 210309_0205_Temp.png
210309_0205_Temp.png
Attachment 6: 210309_0205_Bias.png
210309_0205_Bias.png
Attachment 7: 210309_0529_layout1.png
210309_0529_layout1.png
Attachment 8: 210309_0619_Stats.png
210309_0619_Stats.png
Attachment 9: 210309_0620_Temp.png
210309_0620_Temp.png
Attachment 10: 210309_0623_Bias.png
210309_0623_Bias.png
  181   Mon Mar 8 17:02:38 2021 CAMonday 8th March 18:00 - 00:00
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

18:24 Rate spectra - attachment 4

18:30 Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

19:28 beam back after a brief outage - AIDA on file R9_736

      FRS having DAQ issues

19:57 FRS DAQ/Scalars back

20:12 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

      Merger ok - 4.5M data items/s

      TapeServer ok - 46MB/s

      Data forwarding to MBS ok

20:33 No beam - AIDA on file R9_821

21:08 rates spectra - attachment 8

22:35 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 9
      FEE Temperatures - attachment 10
      Bias and leakage currents ok - attachment 11

      Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

      temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.

23:07 media/ThirdDrive at 65% capacity

      writing to file R9_1021

      file compression currently up to R1_686

23:36 beam stable ~1e9 pps

      rate spectra - attachment 12




      
Attachment 1: Screenshot_from_2021-03-08_17-05-44.png
Screenshot_from_2021-03-08_17-05-44.png
Attachment 2: Screenshot_from_2021-03-08_17-07-04.png
Screenshot_from_2021-03-08_17-07-04.png
Attachment 3: Screenshot_from_2021-03-08_17-08-51.png
Screenshot_from_2021-03-08_17-08-51.png
Attachment 4: Screenshot_from_2021-03-08_17-23-58.png
Screenshot_from_2021-03-08_17-23-58.png
Attachment 5: Screenshot_from_2021-03-08_19-10-56.png
Screenshot_from_2021-03-08_19-10-56.png
Attachment 6: Screenshot_from_2021-03-08_19-11-33.png
Screenshot_from_2021-03-08_19-11-33.png
Attachment 7: Screenshot_from_2021-03-08_19-11-53.png
Screenshot_from_2021-03-08_19-11-53.png
Attachment 8: Screenshot_from_2021-03-08_20-08-12.png
Screenshot_from_2021-03-08_20-08-12.png
Attachment 9: Screenshot_from_2021-03-08_21-28-29.png
Screenshot_from_2021-03-08_21-28-29.png
Attachment 10: Screenshot_from_2021-03-08_21-29-08.png
Screenshot_from_2021-03-08_21-29-08.png
Attachment 11: Screenshot_from_2021-03-08_21-32-35.png
Screenshot_from_2021-03-08_21-32-35.png
Attachment 12: Screenshot_from_2021-03-08_22-38-41.png
Screenshot_from_2021-03-08_22-38-41.png
  180   Mon Mar 8 09:45:20 2021 CA, TDAnalysis R7_20 (new version of NewMerger with min info code 4 & 5 data items)
An analysis of file R6_70 can be found at https://elog.ph.ed.ac.uk/DESPEC/177 attachment 19
We observe c. 80M ADC data items per 2Gb file and c. 160M info code 4 & 5 data items.

This am NH switched NewMerger for a new version which minimises the number of info code 4 & 5
data items. The analysis of one of these data files (R7_20) is appended - attachment 1. We
observe c. 2M ADC data items per 2Gb file. The number of info code 4 data items is significantly
reduced as expected but info code 5 data items are not observed.

Info code 4 & 5 data items will increment every (2^28 * 1e-9) 0.268s and (2^48 * 1e-9) 3.26d.

If we examine the verbose output (attachment 2) we observe that each data block only contains
c. 60 ADC data items - a small fraction of the capacity of each data block.  
Attachment 1: R7_20
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:    1822115 ( 2111176.3 Hz)
        Other data format:       1885 (    2184.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          8 (       9.3 Hz)
                               RESUME:         10 (      11.6 Hz)
                              SYNC100:          4 (       4.6 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:       1857 (    2151.6 Hz)
                             MBS info:          6 (       7.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:        0.863 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.206                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.060                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.201                0.000
  9                0.166                0.805
 10                0.000                0.000
 11                0.592                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     150071         10          0          0          2          2          0          0          0          6          0          0
  1     194013          1          0          0          0          0          1          0          0          0          0          0
  2     150373       1859          0          0          1          1          0          0       1857          0          0          0
  3     230996          0          0          0          0          0          0          0          0          0          0          0
  4      53452          0          0          0          0          0          0          0          0          0          0          0
  5     142131          5          0          0          2          3          0          0          0          0          0          0
  6     178856          0          0          0          0          0          0          0          0          0          0          0
  7      74144          0          0          0          0          0          0          0          0          0          0          0
  8     183174          3          0          0          1          2          0          0          0          0          0          0
  9     218137          5          0          0          1          1          3          0          0          0          0          0
 10      83482          0          0          0          0          0          0          0          0          0          0          0
 11     163286          2          0          0          1          1          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.008s ( 1390.832 blocks/s,  86.927 Mb/s)
Attachment 2: R7_20_verbose
[npg@aidas-gsi analyser]$ ./analyser v  /TapeData/S452/R7_20 | more
 *** TDR format 3.3.0 analyser - TD - January 2019
  verbose
 ***            ADC data: block:     1 ptr:     7 data: 0xC1177F65 module:  4 fail: 0 range: 0 id:   279 channel: 23 adc: 32613  ts: 0x00000000078DEB0E
 ***            ADC data: block:     1 ptr:     9 data: 0xC0478143 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 33091  ts: 0x00000000078DEB4A
 ***            ADC data: block:     1 ptr:    11 data: 0xC0D182FC module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33532  ts: 0x00000000078DEB68
 ***            ADC data: block:     1 ptr:    13 data: 0xC2C081B9 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33209  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    15 data: 0xC2D7820D module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33293  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    17 data: 0xC2E4835E module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33630  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    19 data: 0xC2F681F4 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33268  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    21 data: 0xC00B7FDF module:  0 fail: 0 range: 0 id:    11 channel: 11 adc: 32735  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    23 data: 0xC0147DDE module:  0 fail: 0 range: 0 id:    20 channel: 20 adc: 32222  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    25 data: 0xC034803A module:  0 fail: 0 range: 0 id:    52 channel: 52 adc: 32826  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    27 data: 0xC272814E module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33102  ts: 0x00000000078DF2A2
 ***            ADC data: block:     1 ptr:    29 data: 0xC11B7E13 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32275  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    31 data: 0xC1327D8E module:  4 fail: 0 range: 0 id:   306 channel: 50 adc: 32142  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    33 data: 0xC05F7FB9 module:  1 fail: 0 range: 0 id:    95 channel: 31 adc: 32697  ts: 0x00000000078DF31A
 ***            ADC data: block:     1 ptr:    35 data: 0xC0C28087 module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32903  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    37 data: 0xC0DB8075 module:  3 fail: 0 range: 0 id:   219 channel: 27 adc: 32885  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    39 data: 0xC1BA7ECE module:  6 fail: 0 range: 0 id:   442 channel: 58 adc: 32462  ts: 0x00000000078DF644
 ***            ADC data: block:     1 ptr:    41 data: 0xC2C381FB module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33275  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    43 data: 0xC2D981FD module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33277  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    45 data: 0xC2E682A5 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33445  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    47 data: 0xC2F782E0 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 33504  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    49 data: 0xC0177CE4 module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 31972  ts: 0x00000000078DF982
 ***            ADC data: block:     1 ptr:    51 data: 0xC27A7F8E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32654  ts: 0x00000000078DFA72
 ***            ADC data: block:     1 ptr:    53 data: 0xC0C3816A module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33130  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    55 data: 0xC0FD81FD module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33277  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    57 data: 0xC0847F9A module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32666  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    59 data: 0xC0B77EED module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32493  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    61 data: 0xC1AD8002 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32770  ts: 0x00000000078DFE14
 ***            ADC data: block:     1 ptr:    63 data: 0xC2C4837A module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33658  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    65 data: 0xC2DC8250 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33360  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    67 data: 0xC2EA8214 module: 11 fail: 0 range: 0 id:   746 channel: 42 adc: 33300  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    69 data: 0xC27B812C module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33068  ts: 0x00000000078E0242
 ***            ADC data: block:     1 ptr:    71 data: 0xC0C5811F module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33055  ts: 0x00000000078E02D8
 ***            ADC data: block:     1 ptr:    73 data: 0xC0877F4D module:  2 fail: 0 range: 0 id:   135 channel:  7 adc: 32589  ts: 0x00000000078E0486
 ***            ADC data: block:     1 ptr:    75 data: 0xC2C782D6 module: 11 fail: 0 range: 0 id:   711 channel:  7 adc: 33494  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    77 data: 0xC2DE8261 module: 11 fail: 0 range: 0 id:   734 channel: 30 adc: 33377  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    79 data: 0xC2EB8115 module: 11 fail: 0 range: 0 id:   747 channel: 43 adc: 33045  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    81 data: 0xC0027C33 module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 31795  ts: 0x00000000078E0922
 ***            ADC data: block:     1 ptr:    83 data: 0xC0C68218 module:  3 fail: 0 range: 0 id:   198 channel:  6 adc: 33304  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    85 data: 0xC0D58287 module:  3 fail: 0 range: 0 id:   213 channel: 21 adc: 33415  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    87 data: 0xC08C4EFE module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 20222  ts: 0x00000000078E0C56
 ***            ADC data: block:     1 ptr:    89 data: 0xC2F48203 module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33283  ts: 0x00000000078E0FA8
 ***            ADC data: block:     1 ptr:    91 data: 0xC267809D module:  9 fail: 0 range: 0 id:   615 channel: 39 adc: 32925  ts: 0x00000000078E11E2
 ***            ADC data: block:     1 ptr:    93 data: 0xC044810C module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 33036  ts: 0x00000000078E125A
 ***            ADC data: block:     1 ptr:    95 data: 0xC0C9815B module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 33115  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    97 data: 0xC0DA8174 module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33140  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    99 data: 0xC1827F8D module:  6 fail: 0 range: 0 id:   386 channel:  2 adc: 32653  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   101 data: 0xC1A77E56 module:  6 fail: 0 range: 0 id:   423 channel: 39 adc: 32342  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   103 data: 0xC2FD846E module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33902  ts: 0x00000000078E1778
 ***            ADC data: block:     1 ptr:   105 data: 0xC27081A7 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33191  ts: 0x00000000078E19B2
 ***            ADC data: block:     1 ptr:   107 data: 0xC0CA8171 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 33137  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   109 data: 0xC0DF80F0 module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 33008  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   111 data: 0xC1F28069 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32873  ts: 0x00000000078E1A7A
 ***            ADC data: block:     1 ptr:   113 data: 0xC1837E89 module:  6 fail: 0 range: 0 id:   387 channel:  3 adc: 32393  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   115 data: 0xC1987EC6 module:  6 fail: 0 range: 0 id:   408 channel: 24 adc: 32454  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   117 data: 0xC1B37F93 module:  6 fail: 0 range: 0 id:   435 channel: 51 adc: 32659  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   119 data: 0xC2C28169 module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33129  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     7 data: 0xC2D5830E module: 11 fail: 0 range: 0 id:   725 channel: 21 adc: 33550  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     9 data: 0xC2E18150 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33104  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:    11 data: 0xC0117F5D module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32605  ts: 0x00000000078E2092
 ***            ADC data: block:     2 ptr:    13 data: 0xC27E81BD module:  9 fail: 0 range: 0 id:   638 channel: 62 adc: 33213  ts: 0x00000000078E2182
 ***            ADC data: block:     2 ptr:    15 data: 0xC07D817F module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33151  ts: 0x00000000078E21FA
 ***            ADC data: block:     2 ptr:    17 data: 0xC0CB8125 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33061  ts: 0x00000000078E2218
 ***            ADC data: block:     2 ptr:    19 data: 0xC1C28219 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33305  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    21 data: 0xC1F77FD8 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32728  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    23 data: 0xC08B7FBA module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32698  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    25 data: 0xC09F7E74 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32372  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    27 data: 0xC1867FE8 module:  6 fail: 0 range: 0 id:   390 channel:  6 adc: 32744  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    29 data: 0xC19A7EC3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32451  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    31 data: 0xC2C98098 module: 11 fail: 0 range: 0 id:   713 channel:  9 adc: 32920  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    33 data: 0xC2DA837D module: 11 fail: 0 range: 0 id:   730 channel: 26 adc: 33661  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    35 data: 0xC2E582C0 module: 11 fail: 0 range: 0 id:   741 channel: 37 adc: 33472  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    37 data: 0xC0CC81D9 module:  3 fail: 0 range: 0 id:   204 channel: 12 adc: 33241  ts: 0x00000000078E29E8
 ***            ADC data: block:     2 ptr:    39 data: 0xC187804E module:  6 fail: 0 range: 0 id:   391 channel:  7 adc: 32846  ts: 0x00000000078E2CF4
 ***            ADC data: block:     2 ptr:    41 data: 0xC2DB81B9 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33209  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    43 data: 0xC2E782E0 module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33504  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    45 data: 0xC0CD80DF module:  3 fail: 0 range: 0 id:   205 channel: 13 adc: 32991  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    47 data: 0xC0D380C6 module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32966  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    49 data: 0xC1897EEF module:  6 fail: 0 range: 0 id:   393 channel:  9 adc: 32495  ts: 0x00000000078E34C4
 ***            ADC data: block:     2 ptr:    51 data: 0xC2E98401 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33793  ts: 0x00000000078E36B8
 ***            ADC data: block:     2 ptr:    53 data: 0xC0E08147 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 33095  ts: 0x00000000078E3988
 ***            ADC data: block:     2 ptr:    55 data: 0xC2C881AC module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33196  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    57 data: 0xC2ED8349 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33609  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    59 data: 0xC2F380C2 module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 32962  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    61 data: 0xC1BD7C77 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 31863  ts: 0x00000000078E4464
 ***            ADC data: block:     2 ptr:    63 data: 0xC2D3837F module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33663  ts: 0x00000000078E4658
 ***            ADC data: block:     2 ptr:    65 data: 0xC24F80FC module:  9 fail: 0 range: 0 id:   591 channel: 15 adc: 33020  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    67 data: 0xC26C80FA module:  9 fail: 0 range: 0 id:   620 channel: 44 adc: 33018  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    69 data: 0xC27782CE module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33486  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    71 data: 0xC0C08052 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 32850  ts: 0x00000000078E4928
 ***            ADC data: block:     2 ptr:    73 data: 0xC08E4C1D module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19485  ts: 0x00000000078E4AD6
 ***            ADC data: block:     2 ptr:    75 data: 0xC1817D36 module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 32054  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    77 data: 0xC1BF7FAF module:  6 fail: 0 range: 0 id:   447 channel: 63 adc: 32687  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    79 data: 0xC2E2826E module: 11 fail: 0 range: 0 id:   738 channel: 34 adc: 33390  ts: 0x00000000078E55F8
 ***            ADC data: block:     2 ptr:    81 data: 0xC1C380E1 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 32993  ts: 0x00000000078E58FA
 ***            ADC data: block:     2 ptr:    83 data: 0xC0F78257 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33367  ts: 0x00000000078E6098
 ***            ADC data: block:     2 ptr:    85 data: 0xC2C681D1 module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33233  ts: 0x00000000078E6598
 ***            ADC data: block:     2 ptr:    87 data: 0xC04E8051 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 32849  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    89 data: 0xC0758189 module:  1 fail: 0 range: 0 id:   117 channel: 53 adc: 33161  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    91 data: 0xC0948066 module:  2 fail: 0 range: 0 id:   148 channel: 20 adc: 32870  ts: 0x00000000078E6A16
 ***            ADC data: block:     2 ptr:    93 data: 0xC0057DFD module:  0 fail: 0 range: 0 id:     5 channel:  5 adc: 32253  ts: 0x00000000078E6EB2
 ***            ADC data: block:     2 ptr:    95 data: 0xC2418102 module:  9 fail: 0 range: 0 id:   577 channel:  1 adc: 33026  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    97 data: 0xC26B8030 module:  9 fail: 0 range: 0 id:   619 channel: 43 adc: 32816  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    99 data: 0xC0077C68 module:  0 fail: 0 range: 0 id:     7 channel:  7 adc: 31848  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   101 data: 0xC0107DBC module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32188  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   103 data: 0xC26F8145 module:  9 fail: 0 range: 0 id:   623 channel: 47 adc: 33093  ts: 0x00000000078E7772
 ***            ADC data: block:     2 ptr:   105 data: 0xC0D18120 module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33056  ts: 0x00000000078E7808
 ***            ADC data: block:     2 ptr:   107 data: 0xC2C081F4 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33268  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   109 data: 0xC2D78240 module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33344  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   111 data: 0xC2E48241 module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33345  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   113 data: 0xC0137D35 module:  0 fail: 0 range: 0 id:    19 channel: 19 adc: 32053  ts: 0x00000000078E7E52
 ***            ADC data: block:     2 ptr:   115 data: 0xC2728164 module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33124  ts: 0x00000000078E7F42
 ***            ADC data: block:     2 ptr:   117 data: 0xC0C18052 module:  3 fail: 0 range: 0 id:   193 channel:  1 adc: 32850  ts: 0x00000000078E7FD8
 ***            ADC data: block:     2 ptr:   119 data: 0xC2C38244 module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33348  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     7 data: 0xC2D982DC module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33500  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     9 data: 0xC2E683C3 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33731  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    11 data: 0xC2F68233 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33331  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    13 data: 0xC0177D5C module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 32092  ts: 0x00000000078E8622
 ***            ADC data: block:     3 ptr:    15 data: 0xC0C2803E module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32830  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    17 data: 0xC0FD81FC module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33276  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    19 data: 0xC1AD7F36 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32566  ts: 0x00000000078E8AB4
 ***            ADC data: block:     3 ptr:    21 data: 0xC2F78642 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 34370  ts: 0x00000000078E8CA8
 ***            ADC data: block:     3 ptr:    23 data: 0xC0C3815B module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33115  ts: 0x00000000078E8F78
 ***            ADC data: block:     3 ptr:    25 data: 0xC0847EE9 module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32489  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    27 data: 0xC0B77F35 module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32565  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    29 data: 0xC27B8125 module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33061  ts: 0x00000000078E96B2
 ***            ADC data: block:     3 ptr:    31 data: 0xC08C3881 module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 14465  ts: 0x00000000078E98F6
 ***            ADC data: block:     3 ptr:    33 data: 0xC2C181D1 module: 11 fail: 0 range: 0 id:   705 channel:  1 adc: 33233  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    35 data: 0xC2D480BF module: 11 fail: 0 range: 0 id:   724 channel: 20 adc: 32959  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    37 data: 0xC2648191 module:  9 fail: 0 range: 0 id:   612 channel: 36 adc: 33169  ts: 0x00000000078E9E82
 ***            ADC data: block:     3 ptr:    39 data: 0xC04780C3 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 32963  ts: 0x00000000078E9EFA
 ***            ADC data: block:     3 ptr:    41 data: 0xC0E6809B module:  3 fail: 0 range: 0 id:   230 channel: 38 adc: 32923  ts: 0x00000000078E9F18
 ***            ADC data: block:     3 ptr:    43 data: 0xC2C4834F module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33615  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    45 data: 0xC2DC8132 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33074  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    47 data: 0xC2FD8325 module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33573  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    49 data: 0xC0C581D9 module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33241  ts: 0x00000000078EA6E8
 ***            ADC data: block:     3 ptr:    51 data: 0xC2E183D2 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33746  ts: 0x00000000078EABE8
 ***            ADC data: block:     3 ptr:    53 data: 0xC0C9806C module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 32876  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    55 data: 0xC0DA814C module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33100  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    57 data: 0xC1C28260 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33376  ts: 0x00000000078EAEEA
 ***            ADC data: block:     3 ptr:    59 data: 0xC08B7FEC module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32748  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    61 data: 0xC09F7EF1 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32497  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    63 data: 0xC2668171 module:  9 fail: 0 range: 0 id:   614 channel: 38 adc: 33137  ts: 0x00000000078EB5F2
 ***            ADC data: block:     3 ptr:    65 data: 0xC0CA8094 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 32916  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    67 data: 0xC0DF806C module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 32876  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    69 data: 0xC1F782AB module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33451  ts: 0x00000000078EB6BA
 ***            ADC data: block:     3 ptr:    71 data: 0xC2C282CB module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33483  ts: 0x00000000078EBB88
 ***            ADC data: block:     3 ptr:    73 data: 0xC07D8111 module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33041  ts: 0x00000000078EBE3A
 ***            ADC data: block:     3 ptr:    75 data: 0xC0CB81B8 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33208  ts: 0x00000000078EBE58
 ***            ADC data: block:     3 ptr:    77 data: 0xC2E98262 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33378  ts: 0x00000000078EC358
 ***            ADC data: block:     3 ptr:    79 data: 0xC044807F module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 32895  ts: 0x00000000078EC60A
 ***            ADC data: block:     3 ptr:    81 data: 0xC19A7DA3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32163  ts: 0x00000000078EC934
 ***            ADC data: block:     3 ptr:    83 data: 0xC2ED8207 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33287  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    85 data: 0xC2F380FC module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 33020  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    87 data: 0xC2708152 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33106  ts: 0x00000000078ECD62
 ***            ADC data: block:     3 ptr:    89 data: 0xC1B47F47 module:  6 fail: 0 range: 0 id:   436 channel: 52 adc: 32583  ts: 0x00000000078ED104
 ***            ADC data: block:     3 ptr:    91 data: 0xC2C88150 module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33104  ts: 0x00000000078ED2F8
 ***            ADC data: block:     3 ptr:    93 data: 0xC27781B4 module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33204  ts: 0x00000000078ED532
 ***            ADC data: block:     3 ptr:    95 data: 0xC0C081FF module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 33279  ts: 0x00000000078ED5C8
 ***            ADC data: block:     3 ptr:    97 data: 0xC1C780DD module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32989  ts: 0x00000000078ED5FA
 ***            ADC data: block:     3 ptr:    99 data: 0xC1BD7D53 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 32083  ts: 0x00000000078ED8D4
 ***            ADC data: block:     3 ptr:   101 data: 0xC2D38300 module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33536  ts: 0x00000000078EDAC8
 ***            ADC data: block:     3 ptr:   103 data: 0xC0027F2E module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 32558  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   105 data: 0xC0117FD8 module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32728  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   107 data: 0xC2588009 module:  9 fail: 0 range: 0 id:   600 channel: 24 adc: 32777  ts: 0x00000000078EDD02
 ***            ADC data: block:     3 ptr:   109 data: 0xC1CD810E module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33038  ts: 0x00000000078EDDCA
 ***            ADC data: block:     3 ptr:   111 data: 0xC1927F0C module:  6 fail: 0 range: 0 id:   402 channel: 18 adc: 32524  ts: 0x00000000078EE0A4
 ***            ADC data: block:     3 ptr:   113 data: 0xC2DB8197 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33175  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   115 data: 0xC2E7821D module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33309  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   117 data: 0xC0097F87 module:  0 fail: 0 range: 0 id:     9 channel:  9 adc: 32647  ts: 0x00000000078EE3E2
 ***            ADC data: block:     3 ptr:   119 data: 0xC2598015 module:  9 fail: 0 range: 0 id:   601 channel: 25 adc: 32789  ts: 0x00000000078EE4D2
 ***            ADC data: block:     4 ptr:     7 data: 0xC0E080A6 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 32934  ts: 0x00000000078EE568
 ***            ADC data: block:     4 ptr:     9 data: 0xC08E4C7E module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19582  ts: 0x00000000078EE716
 ***            ADC data: block:     4 ptr:    11 data: 0xC2E88140 module: 11 fail: 0 range: 0 id:   744 channel: 40 adc: 33088  ts: 0x00000000078EEA68
 ***            ADC data: block:     4 ptr:    13 data: 0xC279812E module:  9 fail: 0 range: 0 id:   633 channel: 57 adc: 33070  ts: 0x00000000078EECA2
 ***            ADC data: block:     4 ptr:    15 data: 0xC0438270 module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 33392  ts: 0x00000000078EED1A
 ***            ADC data: block:     4 ptr:    17 data: 0xC0D280BB module:  3 fail: 0 range: 0 id:   210 channel: 18 adc: 32955  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    19 data: 0xC0F78176 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33142  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    21 data: 0xC1817BAB module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 31659  ts: 0x00000000078EF044
 ***            ADC data: block:     4 ptr:    23 data: 0xC2F481FD module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33277  ts: 0x00000000078EF238
 ***            ADC data: block:     4 ptr:    25 data: 0xC27A7E9E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32414  ts: 0x00000000078EF472
 ***            ADC data: block:     4 ptr:    27 data: 0xC04E80E8 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 33000  ts: 0x00000000078EF4EA
 ***            ADC data: block:     4 ptr:    29 data: 0xC0D3809C module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32924  ts: 0x00000000078EF508
 ***            ADC data: block:     4 ptr:    31 data: 0xC2C6810C module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33036  ts: 0x00000000078EFA08
 ***            ADC data: block:     4 ptr:    33 data: 0xC257808A module:  9 fail: 0 range: 0 id:   599 channel: 23 adc: 32906  ts: 0x00000000078EFC42
 ***            ADC data: block:     4 ptr:    35 data: 0xC0107E1C module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32284  ts: 0x00000000078F0322
 ***            ADC data: block:     4 ptr:    37 data: 0xC25A7FA5 module:  9 fail: 0 range: 0 id:   602 channel: 26 adc: 32677  ts: 0x00000000078F0412

[npg@aidas-gsi analyser]$ 

  179   Mon Mar 8 07:02:17 2021 CA, LSMonday 8th March 08:00 - 17:00
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude 1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

08:20 Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

      Writing to file R6_1148

09:00 Beam off for optimisation - writing to file R6_1212

      Expected back at 09:30

09:30 Beam will be off for a bit longer - an hour or two

09:34 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
      Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds 
to finish
      Something confusing, will look at data and report to VP
      Reverted to original merger and back... 45 MB/s
      R9

12.00 (LS)
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment7)
      FEE temps (attatchment8)
      Rate spectra (attachment9), beam still down
      Leakage Currents written to sheets (attachment10)
      Merger ~45M items/s
      Tapeserver ~43MB/s

14.00 Still no beam but is expected to return soon
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment11)
      FEE temps (attatchment12)
      Leakage Currents written to sheets (attachment13)
      Merger ~45M items/s
      Tapeserver ~44MB/s

14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)

15.00 Beam is back (attachment15)
      Still writing to R9 (roughly around R9_386)

15.28 No beam (attachment16)

15.41 Beam back (attachment17)
      Roughly R9 (roughly around R9_439)

16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
 
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment19)
      Spectra Rate (attachment20)
      FEE temps (attatchment21)
      Leakage Currents written to sheets (attachment22), 
      still rising but looking on Grafana 7 day period should start to lower around 17.00
      Merger ~45M items/s
      Tapeserver ~43MB/s

16.55 Beam down (attachment23), back a couple minutes later

15.51 beam down (attachment24)
      
Attachment 1: Screenshot_from_2021-03-08_07-04-21.png
Screenshot_from_2021-03-08_07-04-21.png
Attachment 2: Screenshot_from_2021-03-08_07-07-24.png
Screenshot_from_2021-03-08_07-07-24.png
Attachment 3: Screenshot_from_2021-03-08_07-07-49.png
Screenshot_from_2021-03-08_07-07-49.png
Attachment 4: Screenshot_from_2021-03-08_08-31-37.png
Screenshot_from_2021-03-08_08-31-37.png
Attachment 5: Screenshot_from_2021-03-08_08-32-58.png
Screenshot_from_2021-03-08_08-32-58.png
Attachment 6: Screenshot_from_2021-03-08_08-34-25.png
Screenshot_from_2021-03-08_08-34-25.png
Attachment 7: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 8: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 11: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 12: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 13: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 14: 1500_Grafana_LeakageCurrents.PNG
1500_Grafana_LeakageCurrents.PNG
Attachment 15: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 16: 1528_SpectraRate.PNG
1528_SpectraRate.PNG
Attachment 17: 1541_SpectraRate.PNG
1541_SpectraRate.PNG
Attachment 18: 1600_R9_469_hit_rates.PNG
1600_R9_469_hit_rates.PNG
Attachment 19: 1620_Statistics.PNG
1620_Statistics.PNG
Attachment 20: 1620_SpectraRate.PNG
1620_SpectraRate.PNG
Attachment 21: 1620_FEETemps.PNG
1620_FEETemps.PNG
Attachment 22: 1620_LeakageCurrents.PNG
1620_LeakageCurrents.PNG
Attachment 23: 1655_SpectraRate.PNG
1655_SpectraRate.PNG
Attachment 24: 1749_SpectraRate.PNG
1749_SpectraRate.PNG
  178   Sun Mar 7 23:13:50 2021 OHMonday 8th March
00:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


00:14 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4

02:15 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped

04:00 Noticed more bad merge events
      System wide checks has also noted an FPGA error.
      All other system wide checks same results as previous

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A3F29 BA0DF4D2 , WR/10=>   23DD31DC5CE3215, Readout Time =>   23DD31DC6F34000 
aida02 : White Rabbit=>  166A3F29 CC49ECD7 , WR/10=>   23DD31DC7A0FE15, Readout Time =>   23DD31DC8428000 
aida03 : White Rabbit=>  166A3F29 DF55B7E8 , WR/10=>   23DD31DC9889264, Readout Time =>   23DD31DCA85C000 
aida04 : White Rabbit=>  166A3F29 EECF82B7 , WR/10=>   23DD31DCB14C045, Readout Time =>   23DD31DCC2B8000 
aida05 : White Rabbit=>  166A3F29 FED2EFD2 , WR/10=>   23DD31DCCAEB195, Readout Time =>   23DD31DCDC14000 
aida06 : White Rabbit=>  166A3F2A 0EDC023D , WR/10=>   23DD31DCE49336C, Readout Time =>   23DD31DCF204000 
aida07 : White Rabbit=>  166A3F2A 1E8170D3 , WR/10=>   23DD31DCFD9BE7B, Readout Time =>   23DD31DD10A0000 
aida08 : White Rabbit=>  166A3F2A 30448F76 , WR/10=>   23DD31DD1A074BF, Readout Time =>   23DD31DD2A34000 
aida09 : White Rabbit=>  166A3F2A 3FE36E61 , WR/10=>   23DD31DD33057D6, Readout Time =>   23DD31DD3B94000 
aida10 : White Rabbit=>  166A3F2A 4E83A1F7 , WR/10=>   23DD31DD4A6C365, Readout Time =>   23DD31DD46A8000 
aida11 : White Rabbit=>  166A3F2A 5FD19086 , WR/10=>   23DD31DD661C1A7, Readout Time =>   23DD31DD7664000 
aida12 : White Rabbit=>  166A3F2A 6FE06F3D , WR/10=>   23DD31DD7FCD7EC, Readout Time =>   23DD31DD9684000 

04:07 Beam is back AIDA on file R6_815
      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10
      ASIC check ok

      Multiple merge data error events observed

06:04 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 11
      FEE Temperatures - attachment 12
      Bias and leakage currents ok - attachment 13

07:22 Beam noticed dropped but back quickly
Attachment 1: 210308_0014_Stats.png
210308_0014_Stats.png
Attachment 2: 2303108_0015_Temp.png
2303108_0015_Temp.png
Attachment 3: 210308_0016_Bias.png
210308_0016_Bias.png
Attachment 4: 210308_0120_System.png
210308_0120_System.png
Attachment 5: 210308_0213_Stats.png
210308_0213_Stats.png
Attachment 6: 210308_0213_Temp.png
210308_0213_Temp.png
Attachment 7: 210308_0215_Bias.png
210308_0215_Bias.png
Attachment 8: 210308_0403_Stats.png
210308_0403_Stats.png
Attachment 9: 210308_0404_Temp.png
210308_0404_Temp.png
Attachment 10: 210308_04011_Bias.png
210308_04011_Bias.png
Attachment 11: 210308_0602_Stats.png
210308_0602_Stats.png
Attachment 12: 210308_0603_Temp.png
210308_0603_Temp.png
Attachment 13: 210308_0604_Bias.png
210308_0604_Bias.png
  177   Sun Mar 7 07:23:26 2021 CA, LSMarch 7th
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:30 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

      DAQ continues to run - R1_1148

      Merger ok - 4.6M data items/s

      TapeServer ok - 43MB/s

08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots

09:19 zeroed all histograms

      writing to file R1_1204

      FRS on file 72

      DESPEC to add AIDA XY hit patterns to online analysis histograms

09:35 beam stop

09:56 beam back 

      writing to file R1_1249

      rate spectra - attachment 5

09:30 AIDA DAQ crashed - lost connection to aida05

      power cycle and reset

09:40 AIDA ok now, writing to file R2

09:50 LS takes over

11.10 (German time)
      System wide checks produces some fails: 
      FEE64 module aida06 global clocks failed, 6 (screenshot6)
      This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there

      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2 (screenshot7)
      Tried a recalibration for the two FEEs in FADC align & control tab but error still there

      Rest of the checks pass
      Statistics (screenshot8)
      FEE temps okay (screenshot9)
      Leakage currents written to sheets (screenshot10)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

13.00 System wide checks produces same fails as last time: 
      FEE64 module aida06 global clocks failed, 6
      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2
      Rest of the checks are okay

      Statistics (screenshot11)
      Spectra rate (screenshot12)
      FEE temps okay (screenshot13)
      Leakage currents written to sheets (screenshot14)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s
      There is 1.5TB left on current drive

14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
      Slow comparators in FEE9-12 raised to 0xd
      

      

      
      
      



        
Attachment 1: Screenshot_from_2021-03-07_07-30-28.png
Screenshot_from_2021-03-07_07-30-28.png
Attachment 2: Screenshot_from_2021-03-07_07-31-27.png
Screenshot_from_2021-03-07_07-31-27.png
Attachment 3: Screenshot_from_2021-03-07_07-31-55.png
Screenshot_from_2021-03-07_07-31-55.png
Attachment 4: Screenshot_from_2021-03-07_07-53-05.png
Screenshot_from_2021-03-07_07-53-05.png
Attachment 5: Screenshot_from_2021-03-07_08-59-23.png
Screenshot_from_2021-03-07_08-59-23.png
Attachment 6: 1110_Systemchecks.PNG
1110_Systemchecks.PNG
Attachment 7: 1110_Systemchecks2.PNG
1110_Systemchecks2.PNG
Attachment 8: 1110_Statistics.PNG
1110_Statistics.PNG
Attachment 9: 1110_FEEtemps.PNG
1110_FEEtemps.PNG
Attachment 10: 1110_Leakagecurrents.PNG
1110_Leakagecurrents.PNG
Attachment 11: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 12: 1300_Spectrumrate.PNG
1300_Spectrumrate.PNG
Attachment 13: 1300_FEEtemps.PNG
1300_FEEtemps.PNG
Attachment 14: 1300_Leakagecurrents.PNG
1300_Leakagecurrents.PNG
  176   Sat Mar 6 23:37:22 2021 TDSunday 7 March 00.00-08:00
00.30 UNILAC problem - no beam

00:39 DAQ continues R1_518
      FRS 190W setting 

      ASIC settinfs 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

01:07 all system wide checks OK

      detector biases & leakage currents OK - see attachment 1
      FEE64 temperatures OK - see attachment 2
      good event statistics - OK - see attachment 3
      Merger OK - see attachment 4
      TapeServer OK - see attachment 5

01:20 zero'd all histograms
      ASIC check load

      rate spectra - see attachment 6

01:48 beam return

      run74 (R1_608)

03.05 all system wide checks OK

      detector biases & leakage currents OK - see attachment 7
      FEE64 temperatures OK - see attachment 8
      good event statistics - OK - see attachment 9
      Merger OK - see attachment 10
      TapeServer OK - see attachment 11

     rate spectra - see attachment 12

04.46 run74 ends (R1_841)

05.00 run75 (R1_859)

05:40 detector biases & leakage currents OK - see attachment 13
      FEE64 temperatures OK - see attachment 14
      good event statistics - OK - see attachment 15
      Merger OK - see attachment 16
      TapeServer OK - see attachment 17

     rate spectra - see attachment 18

07:29 run76 (R1_1055)
Attachment 1: 1.png
1.png
Attachment 2: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 3: Screenshot_2021-03-07_Statistics_aidas-gsi.png
Screenshot_2021-03-07_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: 2.png
2.png
Attachment 8: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 9: Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Attachment 10: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Attachment 11: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Attachment 12: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: 3.png
3.png
Attachment 14: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 15: Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Attachment 17: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Attachment 18: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
  175   Sat Mar 6 06:52:06 2021 OH, LSSaturday 6th March
07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us
Attachment 1: Pulser.png
Pulser.png
Attachment 2: 210306_0752_Stats.png
210306_0752_Stats.png
Attachment 3: 210306_0754_Bias.png
210306_0754_Bias.png
Attachment 4: 210306_0958_Stats.png
210306_0958_Stats.png
Attachment 5: 210306_0959_Layout1.png
210306_0959_Layout1.png
Attachment 6: 210306_1000_Temp.png
210306_1000_Temp.png
Attachment 7: 1102_Statistics.PNG
1102_Statistics.PNG
Attachment 8: 1102_SpectrumRate.PNG
1102_SpectrumRate.PNG
Attachment 9: 1102_FEETemps.PNG
1102_FEETemps.PNG
Attachment 10: 1102_FEETemps2.PNG
1102_FEETemps2.PNG
Attachment 11: 1102_FEETemps3.PNG
1102_FEETemps3.PNG
Attachment 12: 1102_LeakageCurrents.PNG
1102_LeakageCurrents.PNG
Attachment 13: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 14: 1300_SpectrumRate.PNG
1300_SpectrumRate.PNG
Attachment 15: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 16: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 17: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 18: 1500_SpectrumRate.PNG
1500_SpectrumRate.PNG
Attachment 19: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 20: 1500_FEETemps2.PNG
1500_FEETemps2.PNG
Attachment 21: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 22: 210306_1726_Stats.png
210306_1726_Stats.png
Attachment 23: 210306_1747_Temp.png
210306_1747_Temp.png
Attachment 24: 210327_1748_Bias.png
210327_1748_Bias.png
Attachment 25: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
Attachment 26: 210306_1946_Layout1.png
210306_1946_Layout1.png
Attachment 27: 210306_2120_Stats.png
210306_2120_Stats.png
Attachment 28: 210306_2121_Temp.png
210306_2121_Temp.png
Attachment 29: 210306_2123_Bias.png
210306_2123_Bias.png
Attachment 30: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
  174   Fri Mar 5 23:45:41 2021 TDS452 information
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S452 shift checks - attachment 1

S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766

Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort

DESPEC Zoom meeting

Join Zoom Meeting
https://gsi-fair.zoom.us/j/98980822569

Meeting ID: 989 8082 2569
Passcode: S452

FRS Zoom meeting 

Meeting-ID: 973 4809 5114

password: FRS-BT2021

Kenncode: 456657
Attachment 1: S452__Shifts_checkpoints_v1_(1).pdf
S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf
Attachment 2: S452_Shift_Plan.pdf
S452_Shift_Plan.pdf
  173   Fri Mar 5 23:21:59 2021 CA, OHMarch 6th 2021 00:00 - 08:00 shift
00:23 206Hg setting - No beam in AIDA yet

      AIDA not writing to file

      Data being forwarded to MBS

      Fragments expected tonight (?)

00:45 GSI DAQ restart

      FEE64 Temps ok - attachment 1

      All system wide checks ok

01:00 Good event statistics - attachment 2

01:48 detector biases/ leakage currents - ok - attachment 3

02:10 beam on, plug removed

      beam delivered to AIDA

      rates spectra - attachment 4

      peaks at ~ 10 counts/channel in DSSD1 c.2000Hz

02:20 beam stop

02:26 beam restart - rate spectra - attachment 5

02:43 high energy channel spectra - attachment 6

      saved the high energy spectra to layout 2 in spectrum browser

03:00 beam stop

03:08 temperatures ok - attachment 7

      statistics ok - attachment 8

      bias/leakage currents ok - attachment 9

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

      all system wide checks ok

04:24 beam is back

      rate spectra - attachment 10

04:51 206Hg implanting in DSSD2

      Rate spectra - attachment 11

      HEC spectra - attachment 12

05:08 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias/leakage currents ok - attachment 15

      All system wide checks ok

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok 

05:45 beam stopped

05:50 beam back

06:45 beam running, but no implants in AIDA

      possible issues with beam delivery to S4

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

07:10 temperatures ok - attachment 16

      statistics ok - attachment 17

      detector bias/leakage currents ok - attachment 18

      all system wide checks OK














      

      

      
Attachment 1: Screenshot_from_2021-03-06_00-00-27.png
Screenshot_from_2021-03-06_00-00-27.png
Attachment 2: Screenshot_from_2021-03-06_00-05-18.png
Screenshot_from_2021-03-06_00-05-18.png
Attachment 3: Screenshot_from_2021-03-06_00-47-41.png
Screenshot_from_2021-03-06_00-47-41.png
Attachment 4: Screenshot_from_2021-03-06_01-17-12.png
Screenshot_from_2021-03-06_01-17-12.png
Attachment 5: Screenshot_from_2021-03-06_01-25-51.png
Screenshot_from_2021-03-06_01-25-51.png
Attachment 6: Screenshot_from_2021-03-06_01-42-58.png
Screenshot_from_2021-03-06_01-42-58.png
Attachment 7: Screenshot_from_2021-03-06_02-05-46.png
Screenshot_from_2021-03-06_02-05-46.png
Attachment 8: Screenshot_from_2021-03-06_02-06-32.png
Screenshot_from_2021-03-06_02-06-32.png
Attachment 9: Screenshot_from_2021-03-06_02-07-47.png
Screenshot_from_2021-03-06_02-07-47.png
Attachment 10: Screenshot_from_2021-03-06_03-24-28.png
Screenshot_from_2021-03-06_03-24-28.png
Attachment 11: Screenshot_from_2021-03-06_03-53-35.png
Screenshot_from_2021-03-06_03-53-35.png
Attachment 12: Screenshot_from_2021-03-06_03-54-17_-_1.png
Screenshot_from_2021-03-06_03-54-17_-_1.png
Attachment 13: Screenshot_from_2021-03-06_04-09-25.png
Screenshot_from_2021-03-06_04-09-25.png
Attachment 14: Screenshot_from_2021-03-06_04-10-07.png
Screenshot_from_2021-03-06_04-10-07.png
Attachment 15: Screenshot_from_2021-03-06_04-10-32.png
Screenshot_from_2021-03-06_04-10-32.png
Attachment 16: Screenshot_from_2021-03-06_06-07-19.png
Screenshot_from_2021-03-06_06-07-19.png
Attachment 17: Screenshot_from_2021-03-06_06-08-16.png
Screenshot_from_2021-03-06_06-08-16.png
Attachment 18: Screenshot_from_2021-03-06_06-08-44.png
Screenshot_from_2021-03-06_06-08-44.png
  172   Fri Mar 5 22:53:05 2021 OHThursday 5th of March
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.

The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.

Beam plug is in front of AIDA. This is causing a huge spike in the Ge.
  171   Thu Mar 4 11:16:13 2021 OHFurther cases of possible database issues
During testing for the S452 experiment yesterday we were running through the steps of a restart.

The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.

Checking the folder manually it could be seen that the Options file was there.

At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.

This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.

During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.

At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally.
Attachment 1: ErrorMessage.PNG
ErrorMessage.PNG
Attachment 2: errorMessage.txt
EXPERIMENTS/AIDA/Options/aida04 TS_SYNC_PHASE NOT available: error 0x10004; value TS_SYNC_PHASE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ExtClk NOT available: error 0x10004; value ExtClk does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.shift NOT available: error 0x10004; value Aida.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 MACB_TRIG_MODE NOT available: error 0x10004; value MACB_TRIG_MODE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.offset NOT available: error 0x10004; value Aida.offset does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 RunNumber NOT available: error 0x10004; value RunNumber does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_GroupBase NOT available: error 0x10004; value Aida_GroupBase does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Rate.channels NOT available: error 0x10004; value Rate.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.channels NOT available: error 0x10004; value Stat.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Vchannels NOT available: error 0x10004; value Aida.Vchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC.settings NOT available: error 0x10004; value ASIC.settings does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_D_Enable NOT available: error 0x10004; value Aida_Hist_D_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 WAVE_DMA_HWM NOT available: error 0x10004; value WAVE_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.shift NOT available: error 0x10004; value Stat.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Wchannels NOT available: error 0x10004; value Aida.Wchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_V_Enable NOT available: error 0x10004; value Aida_Hist_V_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Include.Aida NOT available: error 0x10004; value Include.Aida does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.channels NOT available: error 0x10004; value Aida.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataAcqPgm NOT available: error 0x10004; value DataAcqPgm does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_H_Enable NOT available: error 0x10004; value Aida_Hist_H_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataFormat NOT available: error 0x10004; value DataFormat does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC_DMA_HWM NOT available: error 0x10004; value ASIC_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_L_Enable NOT available: error 0x10004; value Aida_Hist_L_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
STATE for aida01 returned with an error
error: SOAP http transport timed out after 20000 ms
NONE
error: SOAP http transport timed out after 20000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
  170   Thu Mar 4 11:03:04 2021 NHS452 Operating Notes
Please note the following "quirks" currently present in S452 that are important

a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?

b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal

c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and running
screen -S caenhv /dev/ttyACM0
if screen was just closed you can do
screen -x caenhv
to reconnect

The pulser ideally should be started the same
screen -S pulser /dev/ttyUSB0
To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typing
display settings
And general commands:
help

The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (
tmux attach -r
) to have it "readonly"

There is a small script on nnrpi2
./check_usbs.sh
to see which ttyUSB is which device
As they can change if the pi is rebooted

d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3

To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing

e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal
cat /MIDAS/TclHttpd/Html/ETape/1.txt

f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect

To start MBS in a terminal
ssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay

Please check the MBS rate command
rate -nst -rst -nev -rev -ndata -rdata -st
That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning

Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action
  169   Thu Mar 4 10:13:48 2021 OHAnyDesk connection instuction
Attachment 1: Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
  168   Wed Mar 3 15:02:19 2021 CAMBS command line startup
/MIDAS/Linux/startup/datarelaymbs 
  167   Sat Feb 27 12:47:57 2021 NHAIDA Rates/Situation
Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.

Figure  1: Good Events
Figure  2: Rate Histograms (Log-y)
Figure  3: Waveforms (Odd)
Figure  4: Waveforms (Even)
Figure  5: Waveforms @ 5MS/s (Odd)
Figure  6: Waveforms @ 5MS/s (Even)
Figure  7: Waveforms with Pulser (Odd)
Figure  8: Waveforms with Pulser (Even)*
Figure  9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*

* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse
Attachment 1: firefox_2021-02-27_13-47-19.png
firefox_2021-02-27_13-47-19.png
Attachment 2: firefox_2021-02-27_13-46-59.png
firefox_2021-02-27_13-46-59.png
Attachment 3: firefox_2021-02-27_13-36-27.png
firefox_2021-02-27_13-36-27.png
Attachment 4: firefox_2021-02-27_13-36-52.png
firefox_2021-02-27_13-36-52.png
Attachment 5: firefox_2021-02-27_13-38-19.png
firefox_2021-02-27_13-38-19.png
Attachment 6: firefox_2021-02-27_13-37-53.png
firefox_2021-02-27_13-37-53.png
Attachment 7: firefox_2021-02-27_13-39-48.png
firefox_2021-02-27_13-39-48.png
Attachment 8: firefox_2021-02-27_13-42-55.png
firefox_2021-02-27_13-42-55.png
Attachment 9: firefox_2021-02-27_13-39-33.png
firefox_2021-02-27_13-39-33.png
Attachment 10: firefox_2021-02-27_13-44-10.png
firefox_2021-02-27_13-44-10.png
  166   Mon Feb 22 18:18:47 2021 NHAIDA Rates/Situation
The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.

(All data below taken with thresholds 0xA for all)

Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"

Leakage currents high but S4 very warm at the moment as roof is still closed.

Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.

Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done

Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)

aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3
Attachment 1: GoodRates.png
GoodRates.png
Attachment 2: ADCRates.png
ADCRates.png
Attachment 3: Rates.png
Rates.png
Attachment 4: Waves_0.png
Waves_0.png
Attachment 5: Waves_10.png
Waves_10.png
Attachment 6: EvenWaves_0.png
EvenWaves_0.png
Attachment 7: EvenWaves_10.png
EvenWaves_10.png
Attachment 8: Grafana.png
Grafana.png
  165   Fri Feb 12 17:31:04 2021 NHBackground run
A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21

Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64

S4 access will be limited. 

I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal.
  164   Sat Feb 6 10:45:55 2021 NHAIDA Status Feb 2021
> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
> 
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
> 
> The system has been powered up and biased successfully. To be monitored over the next few days.
> 
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
> 
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
> 
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
> 
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.

The hose has arrived and the water line will be prepared in my office mid next week.

All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)

A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.

The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.

There will be a DESPEC Dry Run next week Mon-Tue.
Attachment 1: pregrd050222_E.png
pregrd050222_E.png
Attachment 2: pregrd050221_O.png
pregrd050221_O.png
Attachment 3: postgrd050221_O.png
postgrd050221_O.png
Attachment 4: postgrd050221_E.png
postgrd050221_E.png
  163   Mon Feb 1 15:19:07 2021 NHAIDA Status Feb 2021
01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.

For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)

The system has been powered up and biased successfully. To be monitored over the next few days.

A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.

A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...

The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.

Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently
  162   Tue Nov 24 13:10:42 2020 NHWaveforms with frame grounded
AIDA Frame is now grounded to platform (will add photo later)

Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now)
Attachment 1: waves_odd.png
waves_odd.png
Attachment 2: waves_even.png
waves_even.png
Attachment 3: 20201124_164416.jpg
20201124_164416.jpg
Attachment 4: 20201124_164413.jpg
20201124_164413.jpg
  161   Tue Oct 6 15:48:06 2020 NHnnrpi kernel update
FYI:

The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second

Oct  4 15:01:22 raspberrypi kernel: [1030048.603846]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753]                                                                                             

A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly.
  160   Tue Jul 7 08:50:27 2020 NHnnrpi2 does not boot
MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption?
Will investigate if SD card can be checked and repaired
  159   Tue Jul 7 08:04:57 2020 NHPower failure in Messhuette
Last week there was a power failure in Messhuette (FRS control room) - AIDA-3 and MBS were affected.
AIDA itself and S4 equipment is on UPS and was not shut down.

AIDA-3 has been restarted and MIDAS restarted. All seems OK

S4 conditions: 24.2 C / 33.5% RH / Td = 7.1 C 
  157   Wed Mar 18 18:02:37 2020 PJCSNew Firmware loaded in all FEE64s

Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.

 

 

  156   Mon Mar 16 07:25:24 2020 NHEnd of Experiment

08:25 Beam Stopped, end of S480

Tape Server stopped at R12_2097

Started again in no storage mode

Will leave compressing runs

  155   Mon Mar 16 05:08:12 2020 TA, AM, ESRegular entry
02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at around 3M items/sec
      tape service running at 26MB/sec

 

All attached to this entry with screenshots to 1-7

Attachment 1: 16__1.png
16__1.png
Attachment 2: 16__2.png
16__2.png
Attachment 3: 16__3.png
16__3.png
Attachment 4: 16__4.png
16__4.png
Attachment 5: 16__5.png
16__5.png
Attachment 6: 16__6.png
16__6.png
  154   Mon Mar 16 01:07:29 2020 TA, AM, ESRegular entry
02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

 

All attached to this entry with screenshots to 1-7

 

 

04:05 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

 

All attached tp this entry with screenshots 7-14

Attachment 1: Screenshot-2.png
Screenshot-2.png
Attachment 2: Screenshot-4.png
Screenshot-4.png
Attachment 3: Screenshot-3.png
Screenshot-3.png
Attachment 4: Screenshot-5.png
Screenshot-5.png
Attachment 5: Screenshot-7.png
Screenshot-7.png
Attachment 6: Screenshot.png
Screenshot.png
Attachment 7: 16_1.png
16_1.png
Attachment 8: 16_2.png
16_2.png
Attachment 9: 16_3.png
16_3.png
Attachment 10: 16_4.png
16_4.png
Attachment 11: 16_5.png
16_5.png
Attachment 12: 16_6.png
16_6.png
Attachment 13: 16_7.png
16_7.png
  153   Sun Mar 15 06:45:00 2020 CA, LS, CGSunday 15th March 08:00 - 24:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

07:46 all system wide checks ok

      good event statistics ok - attachment 1
  
      FEE64 temperatures ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:48 merger ok ~ 3 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

08:50 AIDA writing to file SecondDrive/TapeData/S480/R12_1524

09:56 all system wide checks ok

      good event statistics ok - attachment 4

      detector bias/leakage currents ok - attachment 5
  
      FEE64 temperatures ok - attachment 6

12:09 all system wide checks ok


12:13 all system wide checks ok

      good event statistics ok - attachment 7
  
      FEE64 temperatures ok - attachment 8

      detector bias/leakage currents ok, but ramping up - attachment 9


13:32 merger ok ~ 3 million data items/s

      TapeServer ok ~ 23 MB/s

      data forwarding to MBS ok

      Beam seems to be off

13:33 AIDA writing to file R12_1743

*update* checked DESPEC S480 Elog, issues with UNILAC, beam lost at 13:29

13:55 beam back

13:57 AIDA writing to file R12_1759

14:35 all system wide checks ok

      FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11
  
      detector bias/leakage currents ok, but ramping up - attachment 12

15:18 merger ok ~ 2.7 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

      AIDA writing to file R12_1283

16:00 1.8.H high energy spectra - attachments 13 & 14

      hit-rate spectra - attachment 15
 
16.22 all system wide checks okay 
      FEE temperatures okay (attachment 16)
      leakage currents okay and recorded to spreadsheet (attachment 17)
      good event stats okay (attachment 18)
      merger running at 3M items/sec
      tape service running at 26MB/sec

18.00 began compressing R7 files with command: nice -n 10 gzip -v R7_* &
      
      all system wide checks okay 
      FEE temperatures okay (attachment 19)
      leakage currents okay and recorded to spreadsheet (attachment 20)
      good event stats okay (attachment 21)
      merger running at 3M items/sec
      tape service running at 27MB/sec    

20.10 all system wide checks okay 
      FEE temperatures okay (attachment 22)
      leakage currents okay and recorded to spreadsheet, detectors 1 and 2 still rising (attachment 23)
      good event stats okay (attachment 24)
      merger running at 3M items/sec
      tape service running at 25MB/sec

      rates histogram (attachment 25)
      low energy histograms (attachment 26 and 27)
      high energy histograms (attachments 28 and 29)

      
22.07 all system wide checks okay 
      FEE temperatures okay (attachment 30)
      leakage currents okay and recorded to spreadsheet (attachment 31)
      good event stats okay (attachment 32)
      merger running at 3M items/sec
      tape service running at 25MB/sec
      currently on R12_2097

23.56 all system wide checks okay 
      FEE temperatures okay (attachment 33)
      leakage currents okay and recorded to spreadsheet (attachment 34)
      good event stats okay (attachment 35)
      merger running at 3M items/sec
      tape service running at 26MB/sec
      
finishing up now. night shift group have been informed how to check up on AIDA, there is a sheet on the table with relevant information
      
Attachment 1: 200315_0749_stats.png
200315_0749_stats.png
Attachment 2: 200315_0749_temp.png
200315_0749_temp.png
Attachment 3: 200315_0750_bias.png
200315_0750_bias.png
Attachment 4: 200315_0956_stats.png
200315_0956_stats.png
Attachment 5: 200315_0957_bias.png
200315_0957_bias.png
Attachment 6: 200315_0957_temp.png
200315_0957_temp.png
Attachment 7: 200315_1210_stats.png
200315_1210_stats.png
Attachment 8: 200315_1211_temp.png
200315_1211_temp.png
Attachment 9: 200315_1212_bias.png
200315_1212_bias.png
Attachment 10: corrigan_temperature.png
corrigan_temperature.png
Attachment 11: corrigan_stats.png
corrigan_stats.png
Attachment 12: corrigan_bias.png
corrigan_bias.png
Attachment 13: corriganhe1.png
corriganhe1.png
Attachment 14: corriganhe2.png
corriganhe2.png
Attachment 15: 200315_1553_rates.png
200315_1553_rates.png
Attachment 16: 200315_1619_feetemps.png
200315_1619_feetemps.png
Attachment 17: 200315_1620_leakagecurrents.png
200315_1620_leakagecurrents.png
Attachment 18: 200315_1620_goodevents.png
200315_1620_goodevents.png
Attachment 19: 200315_1758_feetemps.png
200315_1758_feetemps.png
Attachment 20: 200315_1759_leakagecurrents.png
200315_1759_leakagecurrents.png
Attachment 21: 200315_1758_goodevents.png
200315_1758_goodevents.png
Attachment 22: 200315_2004_feetemps.png
200315_2004_feetemps.png
Attachment 23: 200315_2007_leakagecurrents#.png
200315_2007_leakagecurrents#.png
Attachment 24: 200315_2006_goodevents.png
200315_2006_goodevents.png
Attachment 25: 200315_2009_rates.png
200315_2009_rates.png
Attachment 26: 200315_2010_lowodd.png
200315_2010_lowodd.png
Attachment 27: 200315_2011_loweven.png
200315_2011_loweven.png
Attachment 28: 200315_2012_highodd.png
200315_2012_highodd.png
Attachment 29: 200315_2014_higheven.png
200315_2014_higheven.png
Attachment 30: 200315_2202_feetemps.png
200315_2202_feetemps.png
Attachment 31: 200315_2203_leakagecurrents.png
200315_2203_leakagecurrents.png
Attachment 32: 200315_2204_goodevents.png
200315_2204_goodevents.png
Attachment 33: 200315_2358_FEETEMPS.png
200315_2358_FEETEMPS.png
Attachment 34: 200315_2358LEAKAGECURRENTS.png
200315_2358LEAKAGECURRENTS.png
Attachment 35: 200315_2359_goodevents.png
200315_2359_goodevents.png
  152   Sat Mar 14 23:09:28 2020 MB, SJ, MRSunday 15th March 0:00-8:00

AIDA ELog filled out by DESPEC night shift members Matthew Brunet (MB), Shaheen Jazrawi (SJ), and Matthias Rudiger (MR).

0:05-0:08 Beam operators altering settings, beam off/sporadic during this period.

0:46
    All system wide checks passed
    Leak currents lower than previous measurement - attachment #1
    Implants look as previously - attachment #2
    Stats about where they have been - attachment #3
    FEE temperatures fine - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer writing about 25 MB/s

 

2:46

      All system wide checks passed
     Leak currents similar to previous - attachment #5 
     Implants look as previously - attachment #6
     Stats about where they have been - attachment #7
     FEE temperatures fine - attachment #8
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s
 

4:49

      All system wide checks passed
     Leak currents similar to previous - attachment #9
     Implants look as previously - attachment #10
     Stats good - attachment #11
     FEE temperatures fine - attachment #12
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s

6:49

      All system wide checks passed
     Leak currents no significant change - attachment #13
     Implants look as previously - attachment #14
     Stats good - attachment #15
     FEE temperatures fine - attachment #16
     Merger fine, 3 Mega items / sec
     TapeServer writing about 25 MB/s

Attachment 1: Matt-15-BiasesA.png
Matt-15-BiasesA.png
Attachment 2: Matt-15-HistosA.png
Matt-15-HistosA.png
Attachment 3: Matt-15-StatsA.png
Matt-15-StatsA.png
Attachment 4: Matt-15-TempsA-.png
Matt-15-TempsA-.png
Attachment 5: Matt-15-BiasesB.png
Matt-15-BiasesB.png
Attachment 6: Matt-15-HistosB.png
Matt-15-HistosB.png
Attachment 7: Matt-15-StatsB.png
Matt-15-StatsB.png
Attachment 8: Matt-15-TempsB.png
Matt-15-TempsB.png
Attachment 9: Matt-15-BiasesC.png
Matt-15-BiasesC.png
Attachment 10: Matt-15-HistosC.png
Matt-15-HistosC.png
Attachment 11: Matt-15-StatsC.png
Matt-15-StatsC.png
Attachment 12: Matt-15-TempsC.png
Matt-15-TempsC.png
Attachment 13: Matt-15-BiasesD.png
Matt-15-BiasesD.png
Attachment 14: Matt-15-HistosD.png
Matt-15-HistosD.png
Attachment 15: Matt-15-StatsD.png
Matt-15-StatsD.png
Attachment 16: Matt-15-TempsD.png
Matt-15-TempsD.png
  151   Sat Mar 14 07:13:59 2020 DK, TD, LS, CGSaturday 14th March 08:00 - 24:00
8:15
    All system wide checks passed
    Leak currents are continuing to creep up - attachment #1
    FEE temperatures fine - attachment #2
    Stats about where they have been - attachment #3
    Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer still writing about 25 MB/s

09.37 analysis of data file S480/R12_443
      deadtime aida09 c. 5%, all other FEE64s <<1%
      See attachment #5

09.54.51
      biases, SSD1 and 2 over 2 uA each now -  attachment #6

11:20
    All  system-wide checks okay
    Merger okay, 3 Mega items / sec
    TapeServer normal, 27 MB/sec
    Biases as attachment #7
    Stats as #8
    Temps as #9
    Histos as #10

14:10
    All system wide checks okay
    Stats #11
    Temps #12
    Implants #13
    Leak currents are getting a bit higher #14

15:30
    All system wide checks passed
    Stats attachment  #15
    Temps attachment  #16
    Implants  attachment #17
    Biases / leak currents attachment  #18
    Merger okay, 3 Mega items / sec
    TapeServer okay, writing 27 MB/s

16.12 all online spectra zero'd

16.31 stat & rate spectra - attachments 19-22
      1.8.L spectra - attachments 23 & 24
      aida03 pulser peak 57 ch FWHM
      1.8.H spectra - attachments 25 & 26
      NewMerger & TapeServer - attachments 27 & 28

18.29

     system wide checks all okay
     FEE temperatures normal (attachment 29)
     Leakage currents okay and recorded to spreadsheet (attachment 30)
     Good event statistics okay (attachment 31)
     Merger running at 3M items/sec
     Tape server running at 25MB/sec

20.25
     system wide checks all okay
     FEE temperatures normal (attachment 32)
     Leakage currents okay and recorded to spreadsheet (attachment 33)
     Good event statistics okay (attachment 34)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

as of 20.39 just finished R12_961

21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
      was down for maybe 5 minutes is back on at 21.40 (attachment 36)
      think between R12_1004 to R12_1008

22.10
     system wide checks all okay
     FEE temperatures normal (attachment 37)
     Leakage currents okay and recorded to spreadsheet (attachment 38)
     Good event statistics okay (attachment 39)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

23.03 someone comes to ask if we see an increase in implantation rate as they do their side
      I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
      FEE temperatures and leakage currents do not seem any different either
      believe they are calling someone about the issue

23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed
Attachment 1: daid-14-biasA.png
daid-14-biasA.png
Attachment 2: daid-14-tempA.png
daid-14-tempA.png
Attachment 3: daid-14-statsA.png
daid-14-statsA.png
Attachment 4: daid-14-histosA.png
daid-14-histosA.png
Attachment 5: R12_443
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   86997080 ( 1109430.4 Hz)
        Other data format:  174986804 ( 2231519.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         29 (       0.4 Hz)
                               RESUME:         29 (       0.4 Hz)
                              SYNC100:   87327942 ( 1113649.7 Hz)
                              WR48-63:   87327942 ( 1113649.7 Hz)
                           FEE64 disc:     330862 (    4219.3 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     102097 (    1302.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       78.416 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.116                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.066                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                4.659                3.647
  9                0.000                0.000
 10                0.112                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3143962    6596287          0          0          1          1    3246749    3246749     102787          0          0      19287
  1   11441532   22943556          0          0          0          0   11461696   11461696      20164          0          0       9627
  2    6435721   13042134          0          0          2          2    6492617    6492617      56896          0          0      12957
  3    3633801    7382655          0          0          0          0    3672152    3672152      38351          0          0      19271
  4    1100939    2276428          0          0          0          0    1125789    1125789      24850          0          0       7339
  5    2632637    5296610          0          0          2          2    2643081    2643081      10444          0          0       4759
  6    1846149    3780603          0          0          0          0    1875584    1875584      29435          0          0      12550
  7    2374464    4801965          0          0          0          0    2392143    2392143      17679          0          0       8964
  8   20263627   40557223          0          0         22         22   20273602   20273602       9975          0          0       2005
  9   14853979   29722517          0          0          0          0   14858832   14858832       4853          0          0       1283
 10    6417731   12865067          0          0          2          2    6427598    6427598       9867          0          0       2254
 11   12852538   25721759          0          0          0          0   12858099   12858099       5561          0          0       1801
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:34518.602s (    0.927 blocks/s,   0.058 Mb/s)
Attachment 6: bias-daid-14B.png
bias-daid-14B.png
Attachment 7: daid-14-biasC.png
daid-14-biasC.png
Attachment 8: daid-14-tempC.png
daid-14-tempC.png
Attachment 9: daid-14-statsC.png
daid-14-statsC.png
Attachment 10: daid-14-histosC.png
daid-14-histosC.png
Attachment 11: daid-14-tempsD.png
daid-14-tempsD.png
Attachment 12: daid-14-statsD.png
daid-14-statsD.png
Attachment 13: daid-14-histosD.png
daid-14-histosD.png
Attachment 14: daid-14-biasD.png
daid-14-biasD.png
Attachment 15: daid-14-statsE.png
daid-14-statsE.png
Attachment 16: daid-14-tempsE.png
daid-14-tempsE.png
Attachment 17: daid-14-histosE.png
daid-14-histosE.png
Attachment 18: daid-14-biasE.png
daid-14-biasE.png
Attachment 19: 1.png
1.png
Attachment 20: 2.png
2.png
Attachment 21: 3.png
3.png
Attachment 22: 4.png
4.png
Attachment 23: 5.png
5.png
Attachment 24: 6.png
6.png
Attachment 25: 7.png
7.png
Attachment 26: 8.png
8.png
Attachment 27: 9.png
9.png
Attachment 28: 10.png
10.png
Attachment 29: 200314_1823_feetemps.png
200314_1823_feetemps.png
Attachment 30: 200314_1827_leakagecurrents.png
200314_1827_leakagecurrents.png
Attachment 31: 200314_1824_goodevents.png
200314_1824_goodevents.png
Attachment 32: 200314_2020_feetemps.png
200314_2020_feetemps.png
Attachment 33: 200314_2023_leakagecurrents.png
200314_2023_leakagecurrents.png
Attachment 34: 200314_2022_goodevents.png
200314_2022_goodevents.png
Attachment 35: 200314_2136_BEAMDOWN.png
200314_2136_BEAMDOWN.png
Attachment 36: 200314_2141_beamback.png
200314_2141_beamback.png
Attachment 37: 200314_2207_feetemps.png
200314_2207_feetemps.png
Attachment 38: 200314_2210_leakagecurrents.png
200314_2210_leakagecurrents.png
Attachment 39: 200314_2208_goodevents.png
200314_2208_goodevents.png
Attachment 40: 200314_2303_rates.png
200314_2303_rates.png
Attachment 41: 200314_2303_goodevents.png
200314_2303_goodevents.png
  150   Fri Mar 13 23:06:39 2020 CA, OHSaturday 14th March 00:00 - 08:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00.00 DAQ momentarily set to NoStorage
      OH changes symbolic link such that AIDA is writing directly to mounted hard-drives ("SecondDrive" and "ThirdDrive")

00.02 DAQ now writing to file again - /media/SecondDrive/TapeData/S480/R12_0

00.30 all system wide checks ok

      FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3
 
00:43 merger ok - writing at ~ 3 million items/s

      TapeServer ok - writing at ~ 27 Mb/s

      Data forwarding to MBS ok

02:14 all system wide checks ok

      FEE64 temperatures ok - attachment 4

      good event statistics ok - attachment 5

      detector bias and leakage currents ok - attachment 6

02:17 Merger ok  ~ 2.7 million items/s

      TapeServer ok  ~ 27 Mb/s

      Data forwarding to MBS ok

02:18 currently writing to file /media/SecondDrive/TapeData/S480/R12_110

02:57 attachments 7 & 8 - high energy channel spectra

      attachment 9 - hit rate spectra

04:21  FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11

      detector bias and leakage currents ok - attachment 12

      system wide checks ok - except WR decoder status (see attachment 13)

04:26 check ASIC control

04:30 system functions -> select required function -> collect all WhiteRabbit status error counters for baseline
      WR decoder status check now ok

04:50 Merger ok  ~ 3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

04:52 currently writing to file /media/SecondDrive/TapeData/S480/R12_231
 
06:11 all system wide checks ok

      FEE64 temperatures ok - attachment 14

      good event statistics ok - attachment 15

      detector bias and leakage currents ok - attachment 16


06:41 Merger ok  ~ 3.3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

06:42 currently writing to file /media/SecondDrive/TapeData/S480/R12_315

07:25 hit-rate spectra - attachment 17

07:36 low energy channel spectra (1.8.L) - attachments 18 & 19



 


      
Attachment 1: 200314_0020_temp.png
200314_0020_temp.png
Attachment 2: 200314_0030_stats.png
200314_0030_stats.png
Attachment 3: 200314_0031_bias.png
200314_0031_bias.png
Attachment 4: 200314_0212_temp.png
200314_0212_temp.png
Attachment 5: 200314_0213_stats.png
200314_0213_stats.png
Attachment 6: 200314_0214_bias.png
200314_0214_bias.png
Attachment 7: even_he.png
even_he.png
Attachment 8: odd_he.png
odd_he.png
Attachment 9: 200314_0254_rates.png
200314_0254_rates.png
Attachment 10: 200314_0416_temp.png
200314_0416_temp.png
Attachment 11: 200314_0417_stats.png
200314_0417_stats.png
Attachment 12: 200314_0418_bias.png
200314_0418_bias.png
Attachment 13: 200314_WRerror.png
200314_WRerror.png
Attachment 14: 200314_0608_temp.png
200314_0608_temp.png
Attachment 15: 200314_0609_stats.png
200314_0609_stats.png
Attachment 16: 200314_0610_bias.png
200314_0610_bias.png
Attachment 17: 200314_0724_rates.png
200314_0724_rates.png
Attachment 18: 200314_0733_18l_odd.png
200314_0733_18l_odd.png
Attachment 19: 200314_0735_18l_even.png
200314_0735_18l_even.png
  149   Fri Mar 13 15:48:11 2020 Friday 13th 16:00 - 24:00OH, TD
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

16:51 Bias and leakage currents ok but starting to increase in DSSD 1 predominantly - attachment 1
      Statistics ok - attachment 2
      FEE temp ok - attachment 3
      System wide checks all ok 
     
16:57 Histograms zeroed

17:17 ASIC control check done

17:40 Layout 1 - attachment 4
      Layouts 3 and 4 - attachments 4 and 5
      Layouts 5 and 6 - attachments 6 and 7

19:06 Earlier in the shift two extra 8TB drives were formatted and mounted
      Named SecondDrive and ThirdDrive
      TapeData folder created on SecondDrive
      Plan is before the end of this shift to move the symbolic link from the filling up drive to the new drive
      Current prediction is drive would otherwise be full by 12:20 tomorrow.
      Playing it safe and switching early

21:57 Bias and leakage ok - attachment 8
      Statistics ok - attachment 9
      FEE Temps ok - attachment 10
Attachment 1: 200313_1649_bias.png
200313_1649_bias.png
Attachment 2: 200313_1652_stats.png
200313_1652_stats.png
Attachment 3: 200313_1653_temp.png
200313_1653_temp.png
Attachment 4: 200313_1738_Layout1.png
200313_1738_Layout1.png
Attachment 5: 200313_1739_layout2.png
200313_1739_layout2.png
Attachment 6: 200313_1740_layout4.png
200313_1740_layout4.png
Attachment 7: 200313_1740_layout5.png
200313_1740_layout5.png
Attachment 8: 200313_1741_layout6.png
200313_1741_layout6.png
Attachment 9: 200313_2155_bias
Attachment 10: 200313_2156_stats.png
200313_2156_stats.png
Attachment 11: 200313_2156_Temps.png
200313_2156_Temps.png
  148   Fri Mar 13 08:01:53 2020 DKFriday 13th 8:00 to 16:00 shift
8:00 - See previous shift's elog entry.  All good.

9:00
    All system wide checks passed
    Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
    Tape Server okay - 27 MB/s
    Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
    Stats as usual - attachment #2
    Temps normal - attachment #3
    Biases and leaks fine -  attachment #4

Disk usage projection
    Space remaining: ~2 TB
    Fill rate: ~2 GB / 1 minute
    -> 1000 minutes =~ 17 hours

Still gzipping from R6_823

10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
     	Kbytes Written:  -1986382400	Kbytes/sec:  26816
        Rate is fixed at that value on update.
        However, actual data files are being updated so data has been coming.
        DK and NH consulted and believe the situation is fine.
        We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.

10:25 Now we are on R10 after start and stop the TapeServer.  Problem is resolved.

11:30
      Implants: attachment #5
      Stats: attachment #6
      Biases / leaks: attachment #7
      Temps: attachment #8
      System wide checks performed, situation normal.
Attachment 1: daid-13-histosA.png
daid-13-histosA.png
Attachment 2: daid-13-statsA.png
daid-13-statsA.png
Attachment 3: daid-13-tempsA.png
daid-13-tempsA.png
Attachment 4: daid-13-biasA.png
daid-13-biasA.png
Attachment 5: daid-13-implantsB.png
daid-13-implantsB.png
Attachment 6: daid-13-statsB.png
daid-13-statsB.png
Attachment 7: daid-13-biasesB.png
daid-13-biasesB.png
Attachment 8: daid-13-tempsB.png
daid-13-tempsB.png
  147   Thu Mar 12 23:44:48 2020 DKPID Plots
Labelled PID plots.  Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.

There was an error in the printouts for Cd isotopes, but the below attachments are corrected.

The Z-depth (color, not nuclear charge) is log scale.
Attachment 1: pid.png
pid.png
Attachment 2: pid.png
pid.png
Attachment 3: pid.pdf
pid.pdf
  146   Thu Mar 12 23:04:24 2020 LS, DKFriday 13th 00:00 to 08:00 shift
   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space
Attachment 1: 200313_0006_feetemps.png
200313_0006_feetemps.png
Attachment 2: 200313_0008_leakcurrents.png
200313_0008_leakcurrents.png
Attachment 3: 200313_0008_goodevents.png
200313_0008_goodevents.png
Attachment 4: 200313_0015_rateshisto.png
200313_0015_rateshisto.png
Attachment 5: 200313_0018_loweoddfees.png
200313_0018_loweoddfees.png
Attachment 6: 200313_0019_loweevenfees.png
200313_0019_loweevenfees.png
Attachment 7: 200313_0021_higheoddfees.png
200313_0021_higheoddfees.png
Attachment 8: 200313_0022_higheevenfees.png
200313_0022_higheevenfees.png
Attachment 9: 200313_0203_feetemps.png
200313_0203_feetemps.png
Attachment 10: 200313_0203_leakagecurrents.png
200313_0203_leakagecurrents.png
Attachment 11: 200313_0205_goodevents.png
200313_0205_goodevents.png
Attachment 12: 200313_0403_feetemps.png
200313_0403_feetemps.png
Attachment 13: 200313_0408_leakagecurrents.png
200313_0408_leakagecurrents.png
Attachment 14: 200313_0406_goodevents.png
200313_0406_goodevents.png
Attachment 15: 200313_0602_feetemps.png
200313_0602_feetemps.png
Attachment 16: 200313_0605_leakagecurrents.png
200313_0605_leakagecurrents.png
Attachment 17: 200313_0605_goodevents.png
200313_0605_goodevents.png
Attachment 18: 200313_0756_feetemps.png
200313_0756_feetemps.png
Attachment 19: 200313_0758_leakagecurrents.png
200313_0758_leakagecurrents.png
Attachment 20: 200313_0758_goodevents.png
200313_0758_goodevents.png
  145   Thu Mar 12 15:25:08 2020 Thursday, March 12 16:00 to midnight shiftCA, DK, LS
16:30
    All system wide checks passed
    Temps shown in attachment #1 
    Histos shown in attachment #2
      NB: Zoomed in, linear scale
    Biases shown in attachment #3
    Stats shown in attachment #4
    Merger rates fine 3 Mega items / sec
    TapeServer looks happy, 25 MB/s

Paddy was asking us to give a rough estimate for the number of implants per spill.
I zoomed in on the rates histogram (attachment #2) and it looks like roughly 5 implants per channel in SSD 1 and SSD2.
So this is 5*128*2=1280.  Note that I don't know the rate window size (e.g., 1 second, 1 spill?)  Probably the window does not catch a full spill.
This might be an over-estimate because an implant in SSD2 may leave a signal in SSD1.
Paddy was getting around 1k or 2k fragments per spill, but to first order it seems consistent.

18:36
   Near R9_372 they slightly opened some FRS slits, as there is a belief we may be blocking some of 94Pd.
   Implant rate seems similar to before, see attachment #5
   Biases did not change more than normal, see attachment #6

20:15
   All system wide checks passed
   Temps shown in attachment #7
   Stats shown in attachment #8
   Biases shown in attchment #9

22:12 
   All system wide checks okay
   Histo #10
   Bias #11
   Stat #12
   Temps #13
Attachment 1: daid-12-tempA.png
daid-12-tempA.png
Attachment 2: daid-12-histosA.png
daid-12-histosA.png
Attachment 3: daid-12-biasesA.png
daid-12-biasesA.png
Attachment 4: daid-12-histosB.png
daid-12-histosB.png
Attachment 5: daid-12-biasB.png
daid-12-biasB.png
Attachment 6: daid-12-tempC.png
daid-12-tempC.png
Attachment 7: daid-12-statsC.png
daid-12-statsC.png
Attachment 8: daid-12-biasC.png
daid-12-biasC.png
Attachment 9: daid-12-histosD.png
daid-12-histosD.png
Attachment 10: daid-12-biasesD.png
daid-12-biasesD.png
Attachment 11: daid-12-statsD.png
daid-12-statsD.png
Attachment 12: daid-12-tempsD.png
daid-12-tempsD.png
  144   Thu Mar 12 08:04:24 2020 OH, CA, TDThursday 12th March 08:00-16:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

09:00 Bias and leakage currents ok - attachment 1
      Statistics ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks all ok

09:08 No beam, not sure exactly how long it has been down.
      FRS team are working on it

09:28 still no beam. issues with SIS. 
      DAQ set to NoStorage mode
      most recent file R7_522

10.27 beam is back
      writing to file R9_0

10.43 attachment 4 - hit rate spectra

11:09 all system wide checks ok
      FEE64 temps ok - attachment 5
      good event statistics ok - attachment 6
      detector bias and leakage currents - attachment 7

11.18 merger ok, ~ 2.7 million data items per second
      TapeServer ok, ~ 25 Mb/s data rate
      Data forwarding to MBS ok

11.20 writing to file R9_39

11.22 beam off

11.24 beam back online (R9_42)

11.41 FRS has reverted to an older setting

13:04 hit rate spectra - attachment 8

13:19 all system wide checks ok

13:30 FEE64 temps ok - attachment 9

      good event statistics ok - attachment 10

      detector bias and leakage currents ok - attachment 11


15:31 good event statistics ok - attachment 13

      detector bias and leakage currents ok - attachment 14

      FEE64 temps ok - attachment 15

      all system wide checks ok

      beam seems to have stopped during 2pm meeting

15.35 TapeServer ok, data rate ~ 23 Mb/s

      Merger ok, rate of 2.5 - 3 million data items per second

      data forwarding to MBS ok

15.38 AIDA writing to file R9_233

16:02 beam back - AIDA writing to file R9_250







      
Attachment 1: 200312_0902_bias.png
200312_0902_bias.png
Attachment 2: 200312_0902_stats.png
200312_0902_stats.png
Attachment 3: 200312_0903_temp.png
200312_0903_temp.png
Attachment 4: 200312_1042_rates.png
200312_1042_rates.png
Attachment 5: 200312_1105_temp.png
200312_1105_temp.png
Attachment 6: 200312_1107_stats.png
200312_1107_stats.png
Attachment 7: 200312_1108_stats.png
200312_1108_stats.png
Attachment 8: 200312_1302_rates.png
200312_1302_rates.png
Attachment 9: 200312_1321_temp.png
200312_1321_temp.png
Attachment 10: 200312_1325_stats.png
200312_1325_stats.png
Attachment 11: 200312_1327_bias.png
200312_1327_bias.png
Attachment 12: 200312_1528_stats.png
200312_1528_stats.png
Attachment 13: 200312_1529_bias.png
200312_1529_bias.png
Attachment 14: 300312_1527_temp.png
300312_1527_temp.png
  143   Wed Mar 11 23:44:27 2020 OHThursday 12th March 00:00-08:000
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00:44 Implantation range in AIDA - attachment 1

01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
      During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
      the AIDA boxes and begin moving files onto that

01:35 System wide checks all ok
      Bias and leakage current ok - attachment 2
      Stats all ok - attachment 3
      FEE temperatures ok - attachment 4

      TapeServer rate around 25-29MB per second
      Merger rate around 2.6-3.2 million items per second

04:11 System wide checks all ok
      Bias and leakage currents ok - attachment 5
      Stats all ok - attachment 6
      FEE temp ok - attachment 7
06:38 System wide checks all ok
      Bias and leakage currents ok - attachment 8
      Stats all ok - attachment 9
      FEE temp ok - attachment 10
Attachment 1: 200311_ImpDis.png
200311_ImpDis.png
Attachment 2: 200312_0135_Bias.png
200312_0135_Bias.png
Attachment 3: 200312_0136_Stats.png
200312_0136_Stats.png
Attachment 4: 200312_0139_FEETemp.png
200312_0139_FEETemp.png
Attachment 5: 200312_0405_BIAS.png
200312_0405_BIAS.png
Attachment 6: 200312_0405_stats.png
200312_0405_stats.png
Attachment 7: 200312_0406_Temp.png
200312_0406_Temp.png
Attachment 8: 200312_0638_bias.png
200312_0638_bias.png
Attachment 9: 200312_0639_stats.png
200312_0639_stats.png
Attachment 10: 200312_0639_temp.png
200312_0639_temp.png
  142   Wed Mar 11 17:21:33 2020 CA, TD, OHWednesday 11 March 16.00-00.00
18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 10.png
10.png
Attachment 10: 11.png
11.png
Attachment 11: 12.png
12.png
Attachment 12: 200311_2035_bias.png
200311_2035_bias.png
Attachment 13: 200311_2034_temps.png
200311_2034_temps.png
Attachment 14: 200311_2034_stats.png
200311_2034_stats.png
Attachment 15: 200311_2233_temp.png
200311_2233_temp.png
Attachment 16: 200311_2234_stats.png
200311_2234_stats.png
Attachment 17: 200311_2235_bias.png
200311_2235_bias.png
Attachment 18: 200311_18h_odd.png
200311_18h_odd.png
Attachment 19: 200311_2245_18h_even.png
200311_2245_18h_even.png
Attachment 20: 200311_2242_rates.png
200311_2242_rates.png
Attachment 21: 200311_2314_bias.png
200311_2314_bias.png
Attachment 22: 200311_2314_stats.png
200311_2314_stats.png
Attachment 23: 201103_2313_temp.png
201103_2313_temp.png
  141   Wed Mar 11 16:29:17 2020 CA, LS, TDslow comparator threshold comparison
17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1

17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2

17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3

17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4
Attachment 1: 100keV_stats.png
100keV_stats.png
Attachment 2: 110keVStats.png
110keVStats.png
Attachment 3: 120kevstats.png
120kevstats.png
Attachment 4: 80kevDSSD2.png
80kevDSSD2.png
  140   Tue Mar 10 23:17:31 2020 DK, LS, PW, TD, CAWednesday 11 March 2020
00:15 Shift change near AIDA R6_475

  We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.

  All system-wide checks passed successfully
  Temperatures ok (aida01 still a bit hot) - see attachment #1
  Stats as normal (SSD3 still noisy) - see attachment #2
  Bias and pulser settings remain sane - see attachment #3
  Writing data to disk at ~26 MB/s as yesterday - see attachment #4
  Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
  Rates look not unreasonable for expected implantation in short time window - see attachment #6

2:26
   We are near R6_574
   All system checks are fine
   Temps are fine - attachment #7
   Stats look as before - attachment #8
   Biases and leaks typical - attachment #9
   Merger fine, 3 x 10^6 items / sec (skip screencap)
   TapeServer still sits near 25 MB/s (skip screencap)
   Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10

2:44 R6_587 or R6_588
   The spills seem to have stopped based on the beeping scalars.  Confirmed with other group there are no scint counts.
   See attachment of AIDA rates where no implants can be seen.  See attachment #11

2:46 Spill beeps heard briefly, but they cut again.  R6_590

2:49 Beam seems to be back R6_592
  Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants.  No problem.  See attachment #12 for implants.

4:35 
  System checks all pass (except expected error on master clock)
  Temps okay - attachment #13
  Biases and leak currents normal - attachment #14
  Stats as before - attachment #15
  Merger going, 3 mega items per sec
  TapeServer going, 25 MB/s

5:59 Beam is gone again, near R6_738.  As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
   See attachment #16 which shows no implants in the rate.


   

Contact phones for Control Room: 2222

We called, they are aware of a problem with the magnets and they are working to resolve it.  So we will await them.

Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.  

We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off

7:14  Beam is still off, but I make some notes
   System wide checks all okay
   Temps as normal (see attachment #17)
   Biases and leak currents okay (see attachment #18)
   Stats as normal (see attachment #19)
   Merger going at 3 mega cps
   TapeServer on, around 25 MB/sec
   Implants not seen in histogram rate as expected (see attachment #20)

7:32 We called the operators to confirm that they don't change anything before calling us first.
     Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.

8.00 shift change at file R6_823
     beam is back on, no settings changed.
     particles per spill is similar to before (2k~3k per spill)
     implants now being seen in rate histogram (attachment 21)

8.32 system wide checks all okay
     FEE temperatures are normal (attachment 22)
     leak currents are okay and have been recorded to sheet (attachment 23)
     good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
     merger rate is running around 3M items/sec
     tape service rate is running around 26MB/sec

9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
     problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
     remaking each histogram and saving layout6 seems to be fixed (attachment 27)

Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
    invoked from within
"SpecDetails 1.8.h"
    (in namespace inscope "::urn:SpectrumService" script line 1)
    invoked from within
====

10.05 beam is being stopped to change some thresholds

10.34 system wide checks all okay
      FEE temperatures are normal (attachment 28)
      leak currents are okay and have been recorded to spreadsheet (attachment 29)
      good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24MB/sec
      implants not seen in histogram rate as expected as beam still down (see attachment 30)

10.42 beam still down but taking data, currently file finishing R6_946
      as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
      have zeroed all histograms so will take screenshots at 11.00

11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks

11.25 beam back file S480/R6_975

12.00 analysis file S480/R6_1001 - attachment 33
      FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
      all HEC data rate c. 1kHz
      all ADC data rate c. 1.2MHz

12.23 system wide checks all okay
      FEE temperatures are normal (attachment 34)
      leak currents are okay and have been recorded to spreadsheet (attachment 35)
      good event stats are okay (attachment 36)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 26MB/sec

13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
      seems to be intermittent. file R6_1051
      
13.10 beam back on, no one was called. currently on file R6_1061

13.22 beam off as going to higher intensity beam. file R6_1069

14.42 system wide checks all okay
      FEE temperatures are normal (attachment 38)
      leak currents are okay and have been recorded to spreadsheet (attachment 39)
      good event stats are okay (attachment 40)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24M/sec
14.48 on file R6_1131

16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
      file R6_1193

16:35 compression of files R6_[200-999] commenced (nice -20)

16:40 system wide checks all okay
      FEE temperatures are normal (attachment 41)
      leak currents are okay and have been recorded to spreadsheet (attachment 42)
      good event stats are okay (attachment 43)
Attachment 1: daid-11-tempsA.png
daid-11-tempsA.png
Attachment 2: daid-11-statsA.png
daid-11-statsA.png
Attachment 3: daid-11-biasA.png
daid-11-biasA.png
Attachment 4: daid-11-TSA.png
daid-11-TSA.png
Attachment 5: daid-11-mergerA.png
daid-11-mergerA.png
Attachment 6: daid-11-implantsA.png
daid-11-implantsA.png
Attachment 7: daid-11-tempB.png
daid-11-tempB.png
Attachment 8: daid-11-statsB.png
daid-11-statsB.png
Attachment 9: daid-11-biasB.png
daid-11-biasB.png
Attachment 10: daid-11-histoB.png
daid-11-histoB.png
Attachment 11: daid-11-histos-beamdrop.png
daid-11-histos-beamdrop.png
Attachment 12: daid-11-beamreturn.png
daid-11-beamreturn.png
Attachment 13: daid-11-tempC.png
daid-11-tempC.png
Attachment 14: daid-11-biasC.png
daid-11-biasC.png
Attachment 15: daid-11-statsC.png
daid-11-statsC.png
Attachment 16: daid-beamstop2.png
daid-beamstop2.png
Attachment 17: daid-11-tempD.png
daid-11-tempD.png
Attachment 18: daid-11-biasD.png
daid-11-biasD.png
Attachment 19: daid-11-statsD.png
daid-11-statsD.png
Attachment 20: daid-11-histosD.png
daid-11-histosD.png
Attachment 21: 200311_804_rateshistograms.png
200311_804_rateshistograms.png
Attachment 22: 200311_28_feetemps.png
200311_28_feetemps.png
Attachment 23: 200311_831_leakcurrents.png
200311_831_leakcurrents.png
Attachment 24: 200311_829_goodevents.png
200311_829_goodevents.png
Attachment 25: 200311_845_highEevenfees.png
200311_845_highEevenfees.png
Attachment 26: daid-11-histos-lowgaineven.png
daid-11-histos-lowgaineven.png
Attachment 27: 200311_919_highEevenfixed.png
200311_919_highEevenfixed.png
Attachment 28: 200311_1030_feetemps.png
200311_1030_feetemps.png
Attachment 29: 200311_1032_leakcurrents.png
200311_1032_leakcurrents.png
Attachment 30: 200311_1031_goodevents.png
200311_1031_goodevents.png
Attachment 31: 200311_1040_ratehistogram.png
200311_1040_ratehistogram.png
Attachment 32: 200311_1107_rates.png
200311_1107_rates.png
Attachment 33: R6_1001
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   87080536 ( 1162937.9 Hz)
        Other data format:  174903340 ( 2335788.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         31 (       0.4 Hz)
                               RESUME:         31 (       0.4 Hz)
                              SYNC100:   87327938 ( 1166241.9 Hz)
                              WR48-63:   87327938 ( 1166241.9 Hz)
                           FEE64 disc:     247402 (    3304.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      73144 (     976.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       74.880 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.144                0.000
  4                0.000                0.000
  5                0.033                0.000
  6                0.000                0.000
  7                0.033                0.000
  8                4.160                3.444
  9                0.111                0.000
 10                0.116                0.000
 11                0.111                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2537909    5321817          0          0          1          1    2619908    2619908      81999          0          0      13572
  1   12374388   24792837          0          0          0          0   12389075   12389075      14687          0          0       7144
  2    5995747   12118373          0          0          0          0    6038040    6038040      42293          0          0       8968
  3    3245929    6571235          0          0          3          3    3272386    3272386      26457          0          0      13205
  4    1018810    2094536          0          0          0          0    1037782    1037782      18972          0          0       5843
  5    3175371    6374045          0          0          1          1    3183138    3183138       7767          0          0       3654
  6    1851746    3764473          0          0          0          0    1872073    1872073      20327          0          0       9037
  7    2503081    5045008          0          0          1          1    2516029    2516029      12948          0          0       6580
  8   19164914   38353829          0          0         20         20   19172901   19172901       7987          0          0       1389
  9   15009332   30028700          0          0          2          2   15012676   15012676       3344          0          0        894
 10    5573107   11166435          0          0          2          2    5579846    5579846       6739          0          0       1608
 11   14630202   29272052          0          0          1          1   14634084   14634084       3882          0          0       1250
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:43185.973s (    0.741 blocks/s,   0.046 Mb/s)
Attachment 34: 200311_1219_feetemps.png
200311_1219_feetemps.png
Attachment 35: 200311_1221_leakcurrents.png
200311_1221_leakcurrents.png
Attachment 36: 200311_1220_goodevents.png
200311_1220_goodevents.png
Attachment 37: 200311_1300_rateshistogram.png
200311_1300_rateshistogram.png
Attachment 38: 200311_1440_FEETEMPS.png
200311_1440_FEETEMPS.png
Attachment 39: 200311_1442_leakcurrents.png
200311_1442_leakcurrents.png
Attachment 40: 200311_1442_GOODEVENTS.png
200311_1442_GOODEVENTS.png
Attachment 41: 200311_1638_temp.png
200311_1638_temp.png
Attachment 42: 200311_1639_bias.png
200311_1639_bias.png
Attachment 43: 200311_1639_stats.png
200311_1639_stats.png
  139   Tue Mar 10 12:36:44 2020 TD[How To] Monitor AIDA
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 1 - 1x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 2 = DSSSD # 3

   Select the terminal and obtain a screenshot (Fn-Alt-PrtScn) of DSSSD #1 = #3 detector
   bias voltages and leakage currents
   Upload screenshots to Elog (elog.ph.ed.ac.uk/DESPEC)
   Update leakage current spreadsheet (https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023)

   If S4 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 3 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H W X' tags

4) Desktop 1 - Firefox web browser - Statistics

   Check 'good event' statistics - screenshot to Elog - compare with previous Elog entries

5) Desktop 1 - Firefox web browser - System Wide Checks

   Check all system wide checks - compare results with porevious Elog entries

6) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~2-3M data items/s) consistent with value observed at start of run

   Observe active spinners in NewMerger and TapeServer server terminal windows adjacent to web browser

   Chweck MBS terminal data rate is around 20000kB/s Bottom right terminal on desktop 2

7) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run
  138   Tue Mar 10 10:16:56 2020 OHAIDA Implant Range Scan
A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19
  137   Tue Mar 10 08:48:07 2020 TD, OH, NH, LS, CA, DK, PJW, RPTuesday 10 March 08.00-16.00
c 09.00 aida03 drops Histogram enable at DAQ GO
        NH determined this was due to Options DB corruption and fixed

09.55 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23 (different from the December ones yesterday which experienced corruption)
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - attachments 4-11
    aida06, 08, 10, 12 re-calibrated OK, all others OK
    aida03 failed WR decoder status - collect all WR errors for baseline - all FEE64s pass WR decoder status
    sync all ASIC clocks using ReSync

merger & TapeServer OK - attachments 12-14

13.29 DAQ to disk file S480/R6

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa
     aida01 LEC/MEC fast comparator 0x64, all other FEE64s 0xff

14.57 fast comparator thresholds LEC/MEC changed to 0xff for all FEEs

15.01 FEE temperatures normal, aida1 virtex temperature still slightly over 65 degrees (attachment 15)
      leakage currents recorded to spreadsheet (attachment 16)
      good events rate similar to previous, high in DSSD3 and seems to be returning to normal in DSSD1 except in one FEE (attachments 17 and 18)
      system wide checks OK
      data rate around 22000 KB per second (attachment 19)
as of 15.12 on file R6_80

attachment 20 is total event stats shows events are being implanted
attachment 21 and 22 show low energy spectrum left most peak is pulser and rightmost is central noise. middle peak in aida1
attachment 23 and 24 show high energy spectrum, aida4 histogram not being produced unsure why.
attachment 25 and 26 show waveforms, only aida3 histogram working

16.00 checked ASIC control, all OK

16.35 spill is now changed to 10s with beam intensity increase by factor of 10. 3000 implants per spill.

16.42 leakage currents recorded to spreadsheet

17.10 system wide checks ok
      fee64 temperatures ok - attachment 27
      detector bias/leakage currents ok - attachment 28
      good event statistics ok - attachment 29

17.15 moved slit at S2 - rate drop to try remove some 96Pd

17.20 writing to file R6_166

17.44 merger/tapeserver ok. Data rate ~ 25000 kb/s

18:10 Fine tuning implantation profile
      Removing 0.25mm Al degrader (60 mg/cm^2) to get 96Pd slightly further in to AIDA.  
      DESPEC is closing the current file, R6_203 was just finished on AIDA on typing this.
      (Maybe the end  of run S480f004_ on DESPEC side but that elog was not updated since 16:40?)
      Decision based on assumption of 6 GeV 96Pd having a range 1 mm in silicon.

19.16 system wide checks ok
      good event statistics ok - attachment 30
      detector bias/leakage currents ok - attachment 31
      fee64 temperatures ok - attachment 32
      
19.39 TapeServer/Merger ok. Writing to file R6_265. Data rate ~ 25000 kb/s

20.07 1.8.H high energy spectra front and back side (attachments 33 + 34)

20.36 S4 timing change, increased by factor of 2

20.38 Merger/TapeServer ok. Data being forwarded to MBS.

20.38 writing to file R6_308 at rate of ~ 25000 kb/s

21:21 system wide checks ok
      good event statistics ok - attachment 35
      fee64 temperatures ok - attachment 36
      detector bias/leakage currents ok - attachment 37

21:57 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_368 at rate of ~26000 kb/s

23:10 system wide checks ok
      good event statistics ok - attachment 38
      fee64 temperatures ok - attachment 39
      detector bias/leakage currents ok - attachment 40

23:24 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_435 at rate of 27000 kb/s

23:42 rate spectra - attachment 41











 
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 11.png
11.png
Attachment 12: 12.png
12.png
Attachment 13: 13.png
13.png
Attachment 14: 14.png
14.png
Attachment 15: 200310_1419_feetemps.png
200310_1419_feetemps.png
Attachment 16: 200310_1451_leakcurrents.png
200310_1451_leakcurrents.png
Attachment 17: 200310_1450_goodevents.png
200310_1450_goodevents.png
Attachment 18: 200309_1506goodevents.png
200309_1506goodevents.png
Attachment 19: 200309_1510_datarate.png
200309_1510_datarate.png
Attachment 20: 200310_1517_stathistograms.png
200310_1517_stathistograms.png
Attachment 21: 200310_1517_lowEoddfees.png
200310_1517_lowEoddfees.png
Attachment 22: 200310_1522_lowEevevfees.png
200310_1522_lowEevevfees.png
Attachment 23: 200310_1523_highEoddfees.png
200310_1523_highEoddfees.png
Attachment 24: 200310_1525_highE_evenfees.png
200310_1525_highE_evenfees.png
Attachment 25: 200310_1529_waveformsoddfees.png
200310_1529_waveformsoddfees.png
Attachment 26: 200310_1530_waveformsevenfees.png
200310_1530_waveformsevenfees.png
Attachment 27: 200310_1707_temp.png
200310_1707_temp.png
Attachment 28: 200310_1708_bias.png
200310_1708_bias.png
Attachment 29: 200320_1708_stats.png
200320_1708_stats.png
Attachment 30: 200310_1915_stats.png
200310_1915_stats.png
Attachment 31: 200310_1916_bias.png
200310_1916_bias.png
Attachment 32: 200310_1919_temp.png
200310_1919_temp.png
Attachment 33: 200310_2000_18h_odd.png
200310_2000_18h_odd.png
Attachment 34: 200310_2005_18h_even.png
200310_2005_18h_even.png
Attachment 35: 200310_2119_stats.png
200310_2119_stats.png
Attachment 36: 200310_2119_temp.png
200310_2119_temp.png
Attachment 37: 200310_2120_bias.png
200310_2120_bias.png
Attachment 38: 200310_2307_temp.png
200310_2307_temp.png
Attachment 39: 200310_2308_stats.png
200310_2308_stats.png
Attachment 40: 200310_2309_bias.png
200310_2309_bias.png
Attachment 41: 200310_2342_rates.png
200310_2342_rates.png
  136   Mon Mar 9 13:50:08 2020 LS, CA, DK09/03/2020 system checks
14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?

15:06 Systems checks fine.
      Master clock failed (no master clock)
      all checks passed
      memory checks all around 38k as normal

      Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees

      Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k

      Attachment 4: leakage currents seem normal at expected values

15:16 Merger working for all FEEs
      tape server is on but no storage
      data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)

ASIC controls checked and are all OK

15.48 low energy pulsar peak FWHM

      fee   FHWM
      1     89.08
      2     132.06
      3     70.73
      4     81.84
      5     62.86
      6     79.92
      7     122.13
      8     73.31
      9     198.39
      10    115.53
      11    100.55
      12    144.04

ALL FEEs in downstream SSD are high

16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)

16:25 included rates, note not to common scale in y (attachment 9)

ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
  Did not seem completely reproducible?
  We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad

Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)


19:15 Update

There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.  
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems

One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space 
  This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)

Now we have brought back up the AIDA systems and should run through the checks

Most system checks look good (except some calibration errors)

Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool

Bias and leak currents attached as #10

Temps as #11

Stats as #12

19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.

21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
      A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
      Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET

21:51 AIDA running to no storage again
Attachment 1: 200309_lowE_spectra.png
200309_lowE_spectra.png
Attachment 2: 200309_FEEtemp.png
200309_FEEtemp.png
Attachment 3: 200309_stats.png
200309_stats.png
Attachment 4: 200309_leakcurrents.png
200309_leakcurrents.png
Attachment 5: 200309_lowE_pulsarpeaksodd.png
200309_lowE_pulsarpeaksodd.png
Attachment 6: 200309_lowE_pulsarpeakseven.png
200309_lowE_pulsarpeakseven.png
Attachment 7: 200309_waveformsodd.png
200309_waveformsodd.png
Attachment 8: 200309_waveformseven.png
200309_waveformseven.png
Attachment 9: 200309_rates.png
200309_rates.png
Attachment 10: biases_after_trip.png
biases_after_trip.png
Attachment 11: temps-after-crash.png
temps-after-crash.png
Attachment 12: stats-after-crash.png
stats-after-crash.png
Attachment 13: ImplantRangeRough.png
ImplantRangeRough.png
  135   Mon Mar 9 12:47:18 2020 NHReport: Unusual AIDA Timestamp Error
Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands

 aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68098c
 aida event c1f6851b 0f68098c | ADC 08:54 L 851b
 event time: 15fa4b164f68098c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1ef8238 0f68115c | ADC 08:47 L 8238
 event time: 15fa4b164f68115c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
 event time: 15fa4b164f68115c
 aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
 aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
 event time: 15fa4b164f681382
 aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
 event time: 15fa4b164f681382
 aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68192c
 aida event c1f9849c 0f68192c | ADC 08:57 L 849c
 event time: 15fa4b164f68192c
 aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
 aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
 event time: 15fa4b164f681f94
 aida event c0568084 04b6854a | ADC 02:22 L 8084
 event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240   506459240             (0 errors)

Unsure if the cause, easily handled by ignoring such events but should be "impossible"
  134   Sat Feb 29 10:56:38 2020 NHAlpha Hit Pattern
Circa 700 GB of data

A few channels in DSSD 1 missing? Otherwise seems OK
Attachment 1: alphas_1d.pdf
alphas_1d.pdf
  133   Tue Feb 25 09:55:59 2020 NHHV Scope Traces
Looking at the HV bias in a scope to see if it is the source of noise especially on the p+n side

Channel 1: HV Core channel 4
Channel 2: HV Braid Channel 4
Math: Ch1 - Ch2

100V bias (100 uA leakage current for 1 MOhm)

No obvious frequency components or large frequency components?
I do not know how this compares to RIKEN or STFC-DL

Updated to put the right frequency range (misread before)

Fig 3-4: Normalish looking data 

Fig 5-6: Examples of noise appearing randomly on math line

Fig 7: 1ms long trace, noise stopped appearing

Update 26.02.2020

Fig 8: Trace with all 4 HV channels off

Fig 9: Trace with only oscilloscope biased

Fig 10: Trace with braids unplugged from all 3 FEE64s (only channel 4 connected)

Things to try:

Joining clean earth & chassis ground?

Checking HV jumper?

Finding 'scope probes and testing NIM socket?
Attachment 1: SCRN0057.PNG
SCRN0057.PNG
Attachment 2: SCRN0054.PNG
SCRN0054.PNG
Attachment 3: SCRN0059.PNG
SCRN0059.PNG
Attachment 4: SCRN0060.PNG
SCRN0060.PNG
Attachment 5: SCRN0071.PNG
SCRN0071.PNG
Attachment 6: SCRN0073.PNG
SCRN0073.PNG
Attachment 7: SCRN0075.PNG
SCRN0075.PNG
Attachment 8: SCRN0078.PNG
SCRN0078.PNG
Attachment 9: SCRN0079.PNG
SCRN0079.PNG
Attachment 10: SCRN0080.PNG
SCRN0080.PNG
  131   Fri Feb 21 12:16:06 2020 TD, NHReport - high - unusual FEE64 crash during startup
Attachment 1: ttyUSB4
06/15:20:21|
ISOL Version 1.00 Date 9th January 2017
06/15:20:21|
Flash base address=FC000000
06/15:20:21|
Set Flash to ASync Mode
06/15:20:21|
XST_SUCCESS
06/15:20:21|
Finished copying zImage to RAM 
06/15:20:21|
06/15:20:21|
Found 0 errors checking kernel image
06/15:20:22|VHDL version number 0X12110103 
06/15:20:22|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
06/15:20:22|
Starting LMK 3200 setup 
06/15:20:22|
06/15:20:22|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
06/15:20:22|
.... SPI Base Address=0x81400000
06/15:20:22|
clk_control_reg=0x4
06/15:20:22|Next step is SPIconfig
06/15:20:23|
Control 32(0x81400000)=0x180
06/15:20:23|
SlaveSel(0x81400000)=0x3
06/15:20:23|
Ctrl(0x81400000)=0xE6
06/15:20:23|
Ctrl(0x81400000)=0x86
06/15:20:23|SPIconfig done now to set up the LMK3200 registers
06/15:20:23|LMK #0 : regInit[0]=0x80000000
06/15:20:23|LMK #0 : regInit[1]=0x10070600
06/15:20:23|LMK #0 : regInit[2]=0x60601
06/15:20:23|LMK #0 : regInit[3]=0x60602
06/15:20:23|LMK #0 : regInit[4]=0x60603
06/15:20:23|LMK #0 : regInit[5]=0x70624
06/15:20:23|LMK #0 : regInit[6]=0x70605
06/15:20:23|LMK #0 : regInit[7]=0x70606
06/15:20:23|LMK #0 : regInit[8]=0x70627
06/15:20:23|LMK #0 : regInit[9]=0x10000908
06/15:20:23|LMK #0 : regInit[10]=0xA0022A09
06/15:20:23|LMK #0 : regInit[11]=0x82800B
06/15:20:23|LMK #0 : regInit[12]=0x28C800D
06/15:20:23|LMK #0 : regInit[13]=0x830020E
06/15:20:23|LMK #0 : regInit[14]=0xC800180F
06/15:20:23|
Calibrate completed at 942 counts
06/15:20:23|
Setting Clock Control =0x0000000B, to set GOE and sync bit
06/15:20:23|
Ctrl @ SPIstop (0x81400000)=0x186
06/15:20:23|
0 delay setup completed at 0 counts
06/15:20:23|
Finished Clock setup LMK03200
06/15:20:23|
completed LMK 3200 setup 
06/15:20:23|
Loaded all four ASICs with default settings 
06/15:20:23|
Setting the ADCs into calibration mode 
06/15:20:23|
06/15:20:23|
Control 32(0x81400400)=0x180
06/15:20:24|
SlaveSel(0x81400400)=0xFF
06/15:20:24|
Ctrl(0x81400400)=0xE6
06/15:20:24|
Ctrl(0x81400400)=0x86
06/15:20:24|
Init : Config of AD9252 SPI ok
06/15:20:24|
06/15:20:24|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
06/15:20:24|
ADCs calibrated
06/15:20:24|
06/15:20:24|
Control 32(0x81400400)=0x186
06/15:20:24|
SlaveSel(0x81400400)=0xFF
06/15:20:24|
Ctrl(0x81400400)=0xE6
06/15:20:24|
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
06/15:20:24|
06/15:20:24|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
06/15:20:24|
06/15:20:24|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
06/15:20:24|
Allocating 0x3b78cc bytes for kernel ...
06/15:20:24|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
06/15:20:27|
06/15:20:27|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
06/15:20:34|
Finalizing device tree... flat tree at 0xbd1300
06/15:20:34|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd8 0x21 
06/15:20:34|Using Xilinx Virtex440 machine description
06/15:20:34|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
06/15:20:34|Zone PFN ranges:
06/15:20:34|  DMA      0x00000000 -> 0x00007000
06/15:20:34|  Normal   0x00007000 -> 0x00007000
06/15:20:34|Movable zone start PFN for each node
06/15:20:34|early_node_map[1] active PFN ranges
06/15:20:34|    0: 0x00000000 -> 0x00007000
06/15:20:34|MMU: Allocated 1088 bytes of context maps for 255 contexts
06/15:20:35|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
06/15:20:35|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
06/15:20:35|PID hash table entries: 512 (order: 9, 2048 bytes)
06/15:20:35|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
06/15:20:35|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
06/15:20:35|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
06/15:20:35|Kernel virtual memory layout:
06/15:20:35|  * 0xffffe000..0xfffff000  : fixmap
06/15:20:35|  * 0xfde00000..0xfe000000  : consistent mem
06/15:20:35|  * 0xfde00000..0xfde00000  : early ioremap
06/15:20:35|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
06/15:20:35|NR_IRQS:512
06/15:20:35|clocksource: timebase mult[a00000] shift[22] registered
06/15:20:35|Console: colour dummy device 80x25
06/15:20:35|Mount-cache hash table entries: 512
06/15:20:35|NET: Registered protocol family 16
06/15:20:35|PCI: Probing PCI hardware
06/15:20:35|bio: create slab <bio-0> at 0
06/15:20:35|NET: Registered protocol family 2
06/15:20:35|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
06/15:20:36|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
06/15:20:36|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
06/15:20:36|TCP: Hash tables configured (established 4096 bind 4096)
06/15:20:36|TCP reno registered
06/15:20:36|NET: Registered protocol family 1
06/15:20:36|ROMFS MTD (C) 2007 Red Hat, Inc.
06/15:20:36|msgmni has been set to 214
06/15:20:36|io scheduler noop registered
06/15:20:36|io scheduler anticipatory registered
06/15:20:36|io scheduler deadline registered
06/15:20:36|io scheduler cfq registered (default)
06/15:20:36|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
06/15:20:36|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
06/15:20:36|console [ttyS0] enabled
06/15:20:36|brd: module loaded
06/15:20:36|loop: module loaded
06/15:20:36|Device Tree Probing 'ethernet'
06/15:20:36|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d8:21
06/15:20:36|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
06/15:20:36|XLlTemac: DCR address: 0x80
06/15:20:36|XLlTemac: buffer descriptor size: 32768 (0x8000)
06/15:20:36|XLlTemac: Allocating DMA descriptors with kmalloc
06/15:20:37|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
06/15:20:37|XTemac: PHY detected at address 7.
06/15:20:37|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
06/15:20:37|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37|Using buffer write method
06/15:20:37|cfi_cmdset_0001: Erase suspend on write enabled
06/15:20:37|cmdlinepart partition parsing not available
06/15:20:37|RedBoot partition parsing not available
06/15:20:37|Creating 5 MTD partitions on "fc000000.flash":
06/15:20:37|0x000000000000-0x000000500000 : "golden_firmware"
06/15:20:37|0x000000500000-0x000000800000 : "golden_kernel"
06/15:20:37|0x000000800000-0x000000d00000 : "user_firmware"
06/15:20:37|0x000000d00000-0x000000fe0000 : "user_kernel"
06/15:20:37|0x000000fe0000-0x000001000000 : "env_variables"
06/15:20:38|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
06/15:20:38|SPI: XIlinx spi: bus number now 32766
06/15:20:38|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
06/15:20:38|SPI: XIlinx spi: bus number now 32765
06/15:20:38|mice: PS/2 mouse device common for all mice
06/15:20:38|Device Tree Probing 'i2c'
06/15:20:38| #0 at 0x81600000 mapped to 0xD1030000, irq=22
06/15:20:38|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
06/15:20:38|TCP cubic registered
06/15:20:38|NET: Registered protocol family 17
06/15:20:38|RPC: Registered udp transport module.
06/15:20:38|RPC: Registered tcp transport module.
06/15:20:38|eth0: XLlTemac: Options: 0x3fa
06/15:20:39|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
06/15:20:39|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
06/15:20:39|eth0: XLlTemac: speed set to 1000Mb/s
06/15:20:41|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
06/15:20:41|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
06/15:20:41|Sending DHCP requests ., OK
06/15:20:42|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.3
06/15:20:42|IP-Config: Complete:
06/15:20:42|     device=eth0, addr=192.168.11.3, mask=255.255.255.0, gw=255.255.255.255,
06/15:20:42|     host=aida03, domain=dl.ac.uk, nis-domain=nuclear.physics,
06/15:20:42|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03
06/15:20:42|Looking up port of RPC 100003/2 on 192.168.11.99
06/15:20:42|Looking up port of RPC 100005/1 on 192.168.11.99
06/15:20:42|VFS: Mounted root (nfs filesystem) on device 0:12.
06/15:20:43|Freeing unused kernel memory: 168k init
06/15:20:43|
INIT: version 2.86 booting

06/15:20:43|Starting sysinit...
06/15:20:44|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
06/15:20:45|		Press 'I' to enter interactive startup.
06/15:20:45|Setting clock  (utc): Thu Sep  6 14:17:26 BST 2018 [  OK  ]

06/15:20:46|Building the cache [  OK  ]

06/15:20:46|Setting hostname aida03:  [  OK  ]

06/15:20:48|Mounting local filesystems:  [  OK  ]

06/15:20:49|Enabling /etc/fstab swaps:  [  OK  ]
06/15:20:52|Finishing sysinit...
06/15:20:53|
INIT: Entering runlevel: 3

06/15:20:55|Entering non-interactive startup
06/15:20:56|FATAL: Module ipv6 not found.
06/15:20:57|Bringing up loopback interface:  [  OK  ]

06/15:20:59|FATAL: Module ipv6 not found.
06/15:20:59|Starting system logger: [  OK  ]

06/15:21:00|Starting kernel logger: [  OK  ]

06/15:21:00|Starting rpcbind: [  OK  ]

06/15:21:08|Mounting NFS filesystems:  [  OK  ]

06/15:21:12|Mounting other filesystems:  [  OK  ]

06/15:21:15|Starting xinetd: [  OK  ]

06/15:21:17|Starting midas:  Starting MIDAS Data Acquisition for aida03
06/15:21:18|xaida: device parameters: base=0x81000000 size=0x200000
06/15:21:24|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
06/15:21:24|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
06/15:21:24|xaida: driver assigned major number 254
06/15:21:24|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
06/15:21:31|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
06/15:21:31|AIDAMEM: aidamem: driver assigned major number 253
06/15:21:31|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
06/15:21:38|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
06/15:21:38|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
06/15:21:38|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
06/15:21:38|Computer Name = aida03; Temp Directory = /tmp/tcl361
06/15:21:42|package limit is not available: can't find package limit
06/15:21:45|Running with default file descriptor limit
06/15:21:45|package setuid is not available: can't find package setuid
06/15:21:46|Running as user 0 group 0
06/15:21:46|
06/15:21:47|AIDA Data Acquisition Program Release 8_57.260418 starting
06/15:21:47|
06/15:21:47|Built without pthreads
06/15:21:47|
06/15:21:47|xaida: open:
06/15:21:48|CreatinAIDAMEM: aidamem_open:
06/15:21:48|g NetVars
06/15:21:48|Output buffer length = 65504; format option = 4; transfer option = 3
06/15:21:48|EB transfer option = 3
06/15:21:48|NetVars created and initialised
06/15:21:48|Statistics thread starting
06/15:21:48|Statistics thread created
06/15:21:48|Stat/Rate creation thread starting
06/15:21:48|Data Acquisition task has PID 375
06/15:21:48|Stat/Rate creation thread created
06/15:21:48|Hit/Rate creation thread starting
06/15:21:48|Hit/Rate creation thread created
06/15:21:48|AIDA Heartbeat thread starting
06/15:21:48|Heartbeat thread created
06/15:21:48|Installing signal handlers
06/15:21:48|Done
06/15:21:48|ModuleNum = 0
06/15:21:48|Aida Initialise complete. AidaExecV8_57: Build 260418. HDL version : 12110103 
06/15:21:48|Spectra table initialised
06/15:21:48|AIDA Data Acquisition now all ready to start
06/15:21:48|SIGBUS, SIGSEGV and SIGPIPE traps setup
06/15:21:48|/debug user "debug" password "qy oblg2zuiw"
06/15:21:55|httpd started on port 8015
06/15:21:55|
06/15:21:55|Cannot use /MIDAS/config/TclHttpd/aida03@8015/startup.tcl
06/15:21:56|Custom startup from /MIDAS/config/TclHttpd/aida03/startup.tcl
06/15:21:56|XAIDA Access package 1.0
06/15:21:57|/XAIDAAccessServer
06/15:21:57|XAD9252 Access package 1.0
06/15:21:57|/XAD9252AccessServer
06/15:21:58|/DataBaseAccessServer
06/15:21:58|[  OK  ]

06/15:21:58|/NetVarService
06/15:21:58|/SigTaskService
06/15:21:58|Loaded MemSasAccess
06/15:21:58|/SpectrumService
06/15:21:58|loading tcl/AIDARunControl.tcl for namespace ::
06/15:21:59|/DataAcquisitionControlServer
06/15:21:59|
06/15:21:59|DefineMessage unknown
06/15:21:59|DENX ELDK version 4.2 build 2008-04-01
06/15:21:59|Linux 2.6.31 on a ppc
06/15:21:59|
06/15:21:59|aida03 login: Run Control Server Implementation for AIDA
06/15:21:59|RunControlServer loaded
06/15:21:59|loading Html/RunControl/implementation.tcl
06/15:21:59|RunControl loaded
06/15:22:00|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
06/15:22:00|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
06/15:22:00|created SigTask registers 0 0  0 0
06/15:22:00|Created UI registers
06/15:22:00|EXPERIMENTS/AIDA/Options/aida03 Stat.offset NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:01|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_W_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:01|EXPERIMENTS/AIDA/Options/aida03 TS_SYNC_PHASE NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 ExtClk NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 Aida.shift NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 MACB_TRIG_MODE NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 Aida.offset NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Aida_GroupBase NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Rate.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Stat.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 WAVE_DMA_HWM NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_D_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 ASIC.settings NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida.Vchannels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida.Wchannels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Stat.shift NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Aida.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Include.Aida NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_V_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 DataFormat NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_H_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 DataAcqPgm NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 ASIC_DMA_HWM NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_L_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|EXPERIMENTS/AIDA/Options/aida03 Index NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|AIDA RunControl loaded
06/15:22:07|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
06/15:22:07|
06/16:01:02|
ISOL Version 1.00 Date 9th January 2017
06/16:01:02|
Flash base address=FC000000
06/16:01:02|
Set Flash to ASync Mode
06/16:01:02|
XST_SUCCESS
06/16:01:02|
Finished copying zImage to RAM 
06/16:01:02|
06/16:01:02|
Found 0 errors checking kernel image
06/16:01:03|VHDL version number 0X12110103 
06/16:01:03|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
06/16:01:04|
Starting LMK 3200 setup 
06/16:01:04|
... 127179 more lines ...
  130   Fri Feb 21 09:39:38 2020 TD, NHFriday 21 February
Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK

10:40 ASIC check load x2


13.12 DAQ stopped c. 10.50
      unable to connect messages for multiple FEE64s

      After lunch powwer-cycled FEE64s and restarted DAQ
      aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root

      Data file 20FEB20/R1
      ASIC settings 2019Oct31-13.24.23
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      continue alpha background run
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
  129   Fri Feb 21 09:21:10 2020 NH(Early) Alpha Analysis
Overnight alpha data

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Feb 20 17:46:47 2020
Last event : Fri Feb 21 10:07:57 2020
Duration   : 58869.9 seconds
--------------------------------------
MBS Events   (M):     1248648 (21 Hz)
AIDA Words   (W): 10222680133 (27 kHz)
ADC Words    (A):  3407431088 (-15076 Hz)
-- Decays    (D):  3407017078 (-15083 Hz)
-- Implants  (I):      414010 (7 Hz)
Info Words   (F):  6815249045 (-30146 Hz)
-- SYNC48    (S):  3407559635 (-15074 Hz)
-- SYNC63    (s):  3407559635 (-15074 Hz)
-- PAUSE     (P):         614 (0 Hz)
-- RESUME    (R):         614 (0 Hz)
-- Discrim   (D):      128547 (2 Hz)
-- Unknown   (U):           0 (0 Hz)
Timewarps    (T):          79 (0 Hz)
Dead Time    (X): 26.8658 s (0.04563588%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |           W           A           D           I           F           S           s     P     R       D     U     T  X
   1 |        5025        1246        1093         153        3779        1675        1675     0     0     429     0     0  0
   2 |        6180         991         851         140        5189        2060        2060     0     0    1069     0     0  0
   3 |  7692461404  2564028610  2564028286         324  5128432794  2564153392  2564153392   614   614 	124782     0    58 26.8658
   4 |        2277         642         581          61        1635         759         759     0     0     117     0     0  0
   5 |        6174        1249        1010         239        4925        2058        2058     0     0     809     0     0  0
   6 |     2467062      822244      822191          53     1644818      822354      822354     0     0     110     0     0  0
   7 |        2484         740         701          39        1744         828         828     0     0      88     0     0  0
   8 |   358445004   119481621   119069026      412595   238963383   119481668   119481668     0     0      47     0     1  0
   9 |        7134        1530        1243         287        5604        2378        2378     0     0     848     0     0  0
  10 |  2168031885   722677295   722677295           0  1445354590   722677295   722677295     0     0       0     0    20  0
  11 |        3981        1252        1190          62        2729        1327        1327     0     0      75     0     0  0
  12 |     1241523      413668      413611          57      827855      413841      413841     0     0     173     0     0  0
**** DESPECAnalysis: PostLoop

A few FEEs running "hot", maybe check. 3 had some alignment issues may need rechecking.
Timewarps somehow in aida03 and aida10 but may be code mistake - must check.
Only aida03 ran fast enough to have dead time.

aida10 shows decay data now - DSSD seems ok

Fig 1: DSSD 1D hit pattern 

Fig 2: DSSD Energy

Fig 3: Single channel in aida03 that is hot

Fig 4: Channels 48-64 in aida08 are hot (wiring?)

Fig 5: Channels 17, 19 (crosstalk?) and 27, 33, 38, 39

All otehr channels quiet and OK. 
Attachment 1: DSSD_1d_HP.png
DSSD_1d_HP.png
Attachment 2: DSSD_E.png
DSSD_E.png
Attachment 3: FEE3_43_Hot.png
FEE3_43_Hot.png
Attachment 4: fee8_hot.png
fee8_hot.png
Attachment 5: Fee10_hot.png
Fee10_hot.png
  128   Thu Feb 20 16:35:55 2020 TD, NHThu 20 February 2020
DSSD#3 (most downstream) was replaced

Noted that old DSSD wafer was not sitting properly - detached - in PCB, PCB flexing and/or inadequate epoxy gluing

New DSSD#3 - 2998-22 - 1006 um - 100 V - ~4 uA

Old Detector removed was 3208-14

Fig 1. HV Control

Fig 2. Temps

Fig 3. Stats

Fig 4. Rates

Fig 5-6: Waves

Fig 7-8: Pulser Peaks

Start alpha run at 17:48, 20FEB20, R1
 Slow comparator 0x64
 Pulser off

Fig 9 : Alpha stats

Fig 10: Merger stats

Fig 11: Merger main page

Fig 12: Timestamp errors in merger console
Attachment 1: hv.png
hv.png
Attachment 2: temps.png
temps.png
Attachment 3: stats.png
stats.png
Attachment 4: rates.png
rates.png
Attachment 5: waves_even.png
waves_even.png
Attachment 6: waves_odd.png
waves_odd.png
Attachment 7: pulser_even.png
pulser_even.png
Attachment 8: pulser_odd.png
pulser_odd.png
Attachment 9: alpha_stats.png
alpha_stats.png
Attachment 10: merger.png
merger.png
Attachment 11: merger_stats.png
merger_stats.png
Attachment 12: merger_errors.png
merger_errors.png
  127   Wed Feb 19 10:58:29 2020 NH, PJCSReport: aida09 Kernel Panic & Lost WR
> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
> 
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
> 
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>  
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> 
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5 
> 
> Attach 5: Timestamp shown by merger

Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of 
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a 
logical "rollover" of the lower 14 bits of this 10nS timestamp.


Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?


It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.

The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be 
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.

If this occurs again then the system wide check results would be interesting.
  126   Mon Feb 17 09:48:34 2020 NHReport: aida09 Kernel Panic & Lost WR
aida09 crashed over the weekend and automatically rebooted.
After the reboot the WR timestamp sent to the merger is in the future and hence incorrect

Reset/Setup did not fix issue
Sync ASICs did not fix issue
GSI White Rabbit control page shows a correct WR timestamp

Attach 1: ttyUSB12 (aida09 log with kernel panic)
Attach 2: GSI White Rabbit control page
Attach 3: "Collect All WR Timestamps"
Attach 4: RAW Display for aida09
 
WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000

WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
c.f. "GSI page" timestamp starting 0x15F427F5 

Attach 5: Timestamp shown by merger
Attachment 1: ttyUSB12
Attachment 2: GSI_WR.png
GSI_WR.png
Attachment 3: GSI_WR2.png
GSI_WR2.png
Attachment 4: GSI_WR4.png
GSI_WR4.png
Attachment 5: GSI_WR3.png
GSI_WR3.png
  125   Fri Feb 14 12:50:29 2020 NHAIDA Noise 14.02.2020
Electricians changed the grounding for the DESPEC platform to a large pillar that goes deep into soil

Change in grounding was apparently successful at reducing the noise complaints in Cave C (R3B)

AIDA:

Fig 1 & 2: Waveforms (Odd/Even)
Fig 3 & 4: Waveforms Zoomed
Fig 5: Good Event Rate
Fig 6: Merger Rate: 2 M items/sec
Fig 7: HV supply (no change)

There does seem to be a modest improvement (Merger rate down from 3 million, good event rates seem *better*)
(Reminder: AIDA@GSI sends 3 items per each ADC event, so merger rate is higher)
Not sure if this is still where it should stand?
Attachment 1: wavesOdd.png
wavesOdd.png
Attachment 2: wavesEven.png
wavesEven.png
Attachment 3: wavesOddZ.png
wavesOddZ.png
Attachment 4: stats_goodevents.png
stats_goodevents.png
Attachment 5: mergerstats.png
mergerstats.png
Attachment 6: hv.png
hv.png
  124   Wed Jan 29 19:09:51 2020 NHaida10 troubleshooting
> aida10 has no ASIC events when running without pulser
> 
> DSSD ribbon cable was replaced, no change
> 
> 29.01.2020 - change AIDA half adapter board, no change
> 
> Swapped HV (braid) from aida10 to aida12 - no change in leakage current, now testing data...
>  Looks the same (no data) fig 2
> 
> Very hot channel in aida08 as well:
>  - Channel 104 in histogram Stat = 4.7.L  fig 1
> 
> Change FEE64 card?
> Any other checks possible?

when running the pulser in 10 do all the channels respond correctly ?
what happens if you drop the shaper threshold into the noise ?
  122   Wed Jan 29 12:22:13 2020 NHaida10 troubleshooting
aida10 has no ASIC events when running without pulser

DSSD ribbon cable was replaced, no change

29.01.2020 - change AIDA half adapter board, no change

Swapped HV (braid) from aida10 to aida12 - no change in leakage current, now testing data...
 Looks the same (no data) fig 2

Very hot channel in aida08 as well:
 - Channel 104 in histogram Stat = 4.7.L  fig 1

Change FEE64 card?
Any other checks possible?

31.01.2020:

Installed new FEE64 card aida10: d8:80:39:41:ba:89
Updated dhcpd.conf
System boots OK.
returned bias ground to aida10 for symmetry reasons
Firmware updated from 0x12110103 to 0x18430701

Left recording alpha data over the weekend:
See stats in fig 3

DSSD 1 and 2 look excellent (one broken strip in DSSd 1)?

aida10 still no counts?
aida09 looks very... weird as well?
aida11 and aida12 seem ok?

Detector fault?
Attachment 1: aida08-noise.png
aida08-noise.png
Attachment 2: aida10-noitems.png
aida10-noitems.png
Attachment 3: Stats.png
Stats.png
  121   Mon Jan 20 14:01:23 2020 NHAIDA Noise 20.01.2019
> Checking noise
> 
> DESPEC crate entirely offline except:
> VME Crate (for VETAR2)
> AIDA (All)
> 
> Figs 1-2: Waveforms (Odd/Even)
> Figs 3-4: Zoomed In (Odd/Even)
> Fig 5: Good Event Stats
> Fig 6: Merger Rate
> 
> Noise still present, despite nothing on DESPEC.
> 
> Things to check: 
> Turn off VME (kills time stamps but might be OK for waveform test?)
> Check floor PSU under AIDA is offline too 
> 
> Not sure that other FRS systems (e.g. Ion Catcher) is offline - I think RF may be on.

Update with different sampling rates (in order)
10 (200 ns)
100 (2 us)
1000 (20 us)

Note that some waveforms didn't update very reliably
In particular aida11 does not seem right at large sample rates
Attachment 1: Odd_10_Z.png
Odd_10_Z.png
Attachment 2: Even_10_Z.png
Even_10_Z.png
Attachment 3: Odd_100_Z.png
Odd_100_Z.png
Attachment 4: Even_100_Z.png
Even_100_Z.png
Attachment 5: Odd_1000_Z.png
Odd_1000_Z.png
Attachment 6: Even_1000_Z.png
Even_1000_Z.png
  120   Wed Jan 15 14:53:41 2020 NHAIDA Noise 14.01.2019
Checking noise

DESPEC crate entirely offline except:
VME Crate (for VETAR2)
AIDA (All)

Figs 1-2: Waveforms (Odd/Even)
Figs 3-4: Zoomed In (Odd/Even)
Fig 5: Good Event Stats
Fig 6: Merger Rate

Noise still present, despite nothing on DESPEC.

Things to check: 
Turn off VME (kills time stamps but might be OK for waveform test?)
Check floor PSU under AIDA is offline too 

Not sure that other FRS systems (e.g. Ion Catcher) is offline - I think RF may be on.
Attachment 1: Waves_Odd.png
Waves_Odd.png
Attachment 2: Waves_even.png
Waves_even.png
Attachment 3: WavesZ_Odd.png
WavesZ_Odd.png
Attachment 4: WavesZ_Even.png
WavesZ_Even.png
Attachment 5: EvtStas0xA.png
EvtStas0xA.png
Attachment 6: Merger0XA.png
Merger0XA.png
  119   Fri Dec 20 12:04:03 2019 NH, TDFriday 20 December 2019
Fig 1: DSSD Leakage currents ok

Fig 2: FEE Temps ok

Fig 3: aida08 has a few timestamp errors in merger

Fig 4, 5: GSI WR OK

Fig 6: All System wide checks passed 

FIg 7: Stats good

13:11 - Alpha Run Stopped, Final File: R9_6

FEEs powered off, DSSD bias off.
Attachment 1: Screenshot-CAEN_HV_Control.png
Screenshot-CAEN_HV_Control.png
Attachment 2: FeeTemps.png
FeeTemps.png
Attachment 3: mergerTSerrors.png
mergerTSerrors.png
Attachment 4: gsiwr1.png
gsiwr1.png
Attachment 5: gsiwr2.png
gsiwr2.png
Attachment 6: checks.png
checks.png
Attachment 7: stats.png
stats.png
  118   Fri Dec 20 10:19:29 2019 NHAlpha Analysis (Pre Run)
======================================                                                                                                                                                                                                                                        
AIDA Unpacker Analysis                                                                                                                                                                                                                                                        
--------------------------------------                                                                                                                                                                                                                                        
First event: Fri Dec 13 15:34:12 2019                                                                                                                                                                                                                                         
Last event : Tue Dec 17 08:52:14 2019                                                                                                                                                                                                                                         
Duration   : 321482 seconds  =  90 hours
--------------------------------------                                                                                                                                                                                                                                        
MBS Events   (M): 1748810227 (5 kHz)                                                                                                                                                                                                                                          
AIDA Words   (W): 1748072047 (5 kHz)                                                                                                                                                                                                                                          
ADC Words    (A): 785414339 (2 kHz)                                                                                                                                                                                                                                           
-- Decays    (D): 771101846 (2 kHz)                                                                                                                                                                                                                                           
-- Implants  (I):  14312493 (44 Hz)                                                                                                                                                                                                                                           
Info Words   (F): 962657708 (2 kHz)                                                                                                                                                                                                                                           
-- SYNC48    (S): 2012396012 (6 kHz)                                                                                                                                                                                                                                          
-- SYNC63    (s): 2012396012 (6 kHz)                                                                                                                                                                                                                                          
-- PAUSE     (P):   2925640 (9 Hz)                                                                                                                                                                                                                                            
-- RESUME    (R):   2925667 (9 Hz)                                                                                                                                                                                                                                            
-- Discrim   (D): 1226981673 (3 kHz)                                                                                                                                                                                                                                          
-- Unknown   (U):         0 (0 Hz)                                                                                                                                                                                                                                            
Timewarps    (T):         2 (0 Hz)
Dead Time    (X): 4336.67292 s (1.34896%)
======================================

Possibly one bond wire loss in DSSD 1
Possibly a few lost in DSSD 3 (X)
DSSD 2 is OK.

Will update with post run data for additional statistics
Attachment 1: Alphas.png
Alphas.png
Attachment 2: Alphas_E.png
Alphas_E.png
  116   Thu Dec 19 14:45:42 2019 TDFEE64 ASIC settings - different diode link settings
[npg@aidas-gsi 2019Oct31-13.24.23]$ pwd
/MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23
[npg@aidas-gsi 2019Oct31-13.24.23]$ /bin/csh
[npg@aidas-gsi 2019Oct31-13.24.23]$ foreach f (`ls`)
foreach? echo $f
foreach? cat "$f"/CONTENTS | grep .26
foreach? end
aida01
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida02
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida03
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida04
ASIC.2.26 string 0x23
ASIC.3.26 string 0x23
ASIC.1.26 string 0x23
ASIC.4.26 string 0x23
aida05
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida06
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida07
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida08
ASIC.2.26 string 0x23
ASIC.3.26 string 0x23
ASIC.1.26 string 0x23
ASIC.4.26 string 0x23
aida09
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida10
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida11
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida12
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11

This variation appears to be true for previously saved ASIC settings too

2018Aug21-13.17.09
2018Sep06-14.38.28
2019Mar31-16.15.57
2019Oct31-13.24.23


Current ASIC settings for AIDA@RIKEN for nnaida1-nnaida4 use values 0x23 and 0xbf

aidas1> cd 2019Jun13-08.56.35
/MIDAS/DB/EXPERIMENTS/AIDA/2019Jun13-08.56.35
aidas1> ls
CONTENTS  nnaida10  nnaida12  nnaida14  nnaida16  nnaida18  nnaida2   nnaida21  nnaida23  nnaida3  nnaida5  nnaida7  nnaida9
nnaida1   nnaida11  nnaida13  nnaida15  nnaida17  nnaida19  nnaida20  nnaida22  nnaida24  nnaida4  nnaida6  nnaida8
aidas1> cat nnaida1/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0f
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x00
ASIC.2.26 string 0xbf
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x0d
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0x34
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x00
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0xb2
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000000
ASIC.3.21 string 0x0f
WAVE.1 string 0x00000000
ASIC.3.22 string 0x5c
LED.0 string 0x00411ecc
WAVE.2 string 0x00000000
ASIC.3.23 string 0x08
LED.1 string 0x0000ffff
WAVE.3 string 0x00000000
ASIC.3.24 string 0x68
LED.2 string 0x00411ecc
ASIC.3.25 string 0x08
LED.3 string 0x00000000
ASIC.1.10 string 0x00
LED.4 string 0x00411ecc
ASIC.3.0 string 0x04
ASIC.1.11 string 0x0d
ASIC.3.26 string 0xbf
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00411ecc
ASIC.3.2 string 0x06
ASIC.1.13 string 0x34
LED.7 string 0x00000000
ASIC.3.3 string 0x02
ASIC.1.14 string 0x02
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
LED.9 string 0x000000c8
ASIC.1.16 string 0xd2
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0xb2
ASIC.1.19 string 0x5c
ASIC.1.20 string 0x68
ASIC.3.8 string 0x00
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0f
ASIC.1.22 string 0x5c
ASIC.4.10 string 0x00
ASIC.1.23 string 0x08
ASIC.4.11 string 0x0d
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.4.13 string 0x34
ASIC.1.25 string 0x08
ASIC.4.14 string 0x02
ASIC.1.26 string 0xbf
ASIC.4.15 string 0x20
DISC.6 string 0x00000002
ASIC.4.16 string 0xd2
ASIC.4.0 string 0x04
ASIC.4.17 string 0x80
DISC.7 string 0x24800000
ASIC.4.18 string 0xb2
ASIC.4.1 string 0x05
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.20 string 0x68
ASIC.4.21 string 0x0f
ASIC.4.3 string 0x02
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x00
ASIC.4.26 string 0xbf
ASIC.2.11 string 0x0d
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0x34
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0xb2
ASIC.1.2 string 0x06
aidas1> cat nnaida2/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0f
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x00
ASIC.2.26 string 0xbf
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x0d
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0x34
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x00
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0xb2
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0f
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x00411ecc
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
WAVE.3 string 0x00000080
LED.1 string 0x0000ffff
ASIC.3.24 string 0x68
LED.2 string 0x00411ecc
ASIC.3.25 string 0x08
ASIC.1.10 string 0x00
LED.3 string 0x00000000
ASIC.1.11 string 0x0d
ASIC.3.0 string 0x04
LED.4 string 0x00411ecc
ASIC.3.26 string 0xbf
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00411ecc
ASIC.3.2 string 0x06
ASIC.1.13 string 0x34
ASIC.1.14 string 0x02
ASIC.3.3 string 0x02
LED.7 string 0x00000000
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
ASIC.1.16 string 0xd2
LED.9 string 0x000000c8
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0xb2
ASIC.1.20 string 0x68
ASIC.1.19 string 0x5c
ASIC.3.8 string 0x00
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0f
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x0d
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0x34
ASIC.4.14 string 0x02
ASIC.1.26 string 0xbf
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0xb2
ASIC.4.1 string 0x05
ASIC.4.20 string 0x68
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.3 string 0x02
ASIC.4.21 string 0x0f
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x00
ASIC.4.26 string 0xbf
ASIC.2.11 string 0x0d
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0x34
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0xb2
ASIC.1.2 string 0x06
aidas1> cat nnaida3/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0b
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x01
ASIC.2.26 string 0x23
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x02
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0xdb
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x01
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0x30
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0b
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x004021fc
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
WAVE.3 string 0x00000080
LED.1 string 0x0000ffff
ASIC.3.24 string 0x68
LED.2 string 0x004021fc
ASIC.3.25 string 0x08
ASIC.1.10 string 0x00
LED.3 string 0x00000000
ASIC.1.11 string 0x02
ASIC.3.0 string 0x04
LED.4 string 0x004021fc
ASIC.3.26 string 0x23
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x004021fc
ASIC.3.2 string 0x06
ASIC.1.13 string 0xdb
ASIC.1.14 string 0x02
ASIC.3.3 string 0x02
LED.7 string 0x00000000
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
ASIC.1.16 string 0xd2
LED.9 string 0x000000c8
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0x30
ASIC.1.20 string 0x68
ASIC.1.19 string 0x5c
ASIC.3.8 string 0x01
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0b
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x02
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0xdb
ASIC.4.14 string 0x02
ASIC.1.26 string 0x23
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0x30
ASIC.4.1 string 0x05
ASIC.4.20 string 0x68
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.3 string 0x02
ASIC.4.21 string 0x0b
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x01
ASIC.4.26 string 0x23
ASIC.2.11 string 0x02
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0xdb
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0x30
ASIC.1.2 string 0x06
aidas1> cat nnaida4/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0b
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x01
ASIC.2.26 string 0x23
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x02
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0xdb
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x01
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0x30
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0b
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x00402134
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
LED.1 string 0x0000ffff
WAVE.3 string 0x00000080
ASIC.3.24 string 0x68
LED.2 string 0x00402134
ASIC.3.25 string 0x08
LED.3 string 0x00000000
ASIC.1.10 string 0x00
LED.4 string 0x00402134
ASIC.3.0 string 0x04
ASIC.1.11 string 0x02
ASIC.3.26 string 0x23
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00402134
ASIC.3.2 string 0x06
ASIC.1.13 string 0xdb
LED.7 string 0x00000000
ASIC.3.3 string 0x02
ASIC.1.14 string 0x02
ASIC.1.15 string 0x20
LED.8 string 0x000003fc
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
LED.9 string 0x000000c8
ASIC.1.16 string 0xd2
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0x30
ASIC.1.19 string 0x5c
ASIC.1.20 string 0x68
ASIC.3.8 string 0x01
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0b
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x02
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0xdb
ASIC.4.14 string 0x02
ASIC.1.26 string 0x23
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0x30
ASIC.4.1 string 0x05
ASIC.4.2 string 0x06
ASIC.4.20 string 0x68
ASIC.4.19 string 0x5c
ASIC.4.21 string 0x0b
ASIC.4.3 string 0x02
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x01
ASIC.4.26 string 0x23
ASIC.2.11 string 0x02
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0xdb
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0x30
ASIC.1.2 string 0x06

Have modifuied AIDA@DESPEC  ASIC seetings and saved using DB key 2019Dec19-16.19.51
  115   Thu Dec 19 14:26:09 2019 TDBNC PB-5 - Manual
Attachment 1: pb-5manual.pdf
pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf
  114   Thu Dec 19 14:07:38 2019 TDReport - low - aida08 ADC spectrum size 32k cf. 64k channels expcted using layout
Note that aida08 spectrum 1.8.L size is (shown as) 32k channels cf. 64k channels expected using layout - attachment 1 - 5
Layout configuration - attachment 6
Why?

Current AIDA Options are shown below

[npg@aidas-gsi Options]$ pwd
/MIDAS/DB/EXPERIMENTS/AIDA/Options
[npg@aidas-gsi Options]$ csh
[npg@aidas-gsi Options]$ foreach f (`ls`)
foreach? echo $f
foreach? cat "$f"/CONTENTS
foreach? end
aida01
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
Rate.channels string 512
RunNumber string 12
Aida_GroupBase string 1
WAVE_DMA_HWM string 0x0007ffff
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
Aida_Hist_D_Enable string 1
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida02
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable 
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida03
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida04
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Aida_GroupBase string 1
Rate.channels string 512
Stat.channels string 512
RunNumber string 12
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
MERGE.LinksAvailable string 4
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida05
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida06
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida07
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida08
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida09
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida10
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida11
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida12
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: LayOut4.mlf
6
gallery=1 overlap=1 server=aida02 spectrum=1.8.L
gallery=2 overlap=1 server=aida04 spectrum=1.8.L
gallery=3 overlap=1 server=aida06 spectrum=1.8.L
gallery=4 overlap=1 server=aida08 spectrum=1.8.L
gallery=5 overlap=1 server=aida10 spectrum=1.8.L
gallery=6 overlap=1 server=aida12 spectrum=1.8.L
  113   Thu Dec 19 13:09:32 2019 TDMIDAS app launcher command lines
App                          Path

TclHttpd @ 8015 for AIDA     /MIDAS/Linux/startup/HTTPD
TclHttpd @ 8115 for AIDA     /MIDAS/Linux/startup/HTTPD@8115

TapeServer                   /MIDAS/Linux/startup/TapeServer
New Merger for AIDA          /MIDAS/Linux/startup/NewMerger

MBS Relay                    /MIDAS/Linux/startup/datarelaymbs
  112   Thu Dec 19 13:06:02 2019 TD, NHAIDA@DESPEC To Do list
FAIR phase 0 DESPEC experiments are scheduled March-May 2020 see

https://www.gsi.de/en/work/organisation/scientific_boards/user/beam_time.htm

The current (19.12.19) version of the schedule is v22 

https://www.gsi.de/fileadmin/beamtime/2020/BTS2020_v022_all.pdf

AIDA will used for DESPEC experiments S480, S452, S470 & S460

To be done at GSI

- check DSSSD - FEE64 connection using alpha background data collected pre- and post- Dec 19 test (NH)

- investigate system noise (TD, NH et al.)

- replace aida10 adaptor PCB and/or FEE64 (TD, NH et al.)

Other issues requiring the support of Carl, James, Patrick & Vic are 
in order of our ( = TD + NH ) priority - Carl et al. may have a different
view. If so, we should discuss.

- ASIC settings / Options DB 'corruption' issues (VFEP, CW et al.)

- waveform data (PCS, JL et al.)

- FEE64/ASIC sync (PCS, JL et al.)

ELOG V3.1.3-unknown
  111   Thu Dec 19 10:11:59 2019 TDReport - medium - FEE64 panics during boot
Some of the FEE64s aida01 .. aida12 panic during boot 

Frequencies of panics for each FEE64 can be seen in attachment 1

Below is an example of an aida04 panic following a power cycle and an automatic reboot

pi@raspberrypi:~/logs $ ./tail_aida aida04
aida04
ttyUSB7
19/11:07:04|LR [203f5680] 0x203f5680
19/11:07:04|Call Trace:
19/11:07:04|Kernel panic - not syncing: Fatal exception in interrupt
19/11:07:04|Call Trace:
19/11:07:04|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/11:07:04|[c6941e20] [c00345bc] panic+0x94/0x168
19/11:07:04|[c6941e70] [c000bd44] die+0x178/0x18c
19/11:07:04|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
19/11:07:04|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
19/11:07:04|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
19/11:10:05|
Found 0 errors checking kernel image
19/11:10:06|VHDL version number 0X18430701 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
19/11:10:06|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/11:10:06|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/11:10:06|SPIconfig done now to set up the LMK3200 registers
19/11:10:06|LMK #0 : regInit[0]=0x80000000
19/11:10:07|LMK #0 : regInit[1]=0x10070600
19/11:10:07|LMK #0 : regInit[2]=0x60601
19/11:10:07|LMK #0 : regInit[3]=0x60602
19/11:10:07|LMK #0 : regInit[4]=0x60603
19/11:10:07|LMK #0 : regInit[5]=0x70624
19/11:10:07|LMK #0 : regInit[6]=0x70605
19/11:10:07|LMK #0 : regInit[7]=0x70606
19/11:10:07|LMK #0 : regInit[8]=0x70627
19/11:10:07|LMK #0 : regInit[9]=0x10000908
19/11:10:07|LMK #0 : regInit[10]=0xA0022A09
19/11:10:07|LMK #0 : regInit[11]=0x82800B
19/11:10:07|LMK #0 : regInit[12]=0x28C800D
19/11:10:07|LMK #0 : regInit[13]=0x830020E
19/11:10:07|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/11:10:07|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
19/11:10:07|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/11:10:08|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
19/11:10:08|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/11:10:11|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xb7 
19/11:10:17|Using Xilinx Virtex440 machine description
19/11:10:18|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/11:10:18|Zone PFN ranges:
19/11:10:18|  DMA      0x00000000 -> 0x00007000
19/11:10:18|  Normal   0x00007000 -> 0x00007000
19/11:10:18|Movable zone start PFN for each node
19/11:10:18|early_node_map[1] active PFN ranges
19/11:10:18|    0: 0x00000000 -> 0x00007000
19/11:10:18|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/11:10:18|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/11:10:18|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/11:10:18|PID hash table entries: 512 (order: 9, 2048 bytes)
19/11:10:19|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/11:10:19|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/11:10:19|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/11:10:19|Kernel virtual memory layout:
19/11:10:19|  * 0xffffe000..0xfffff000  : fixmap
19/11:10:19|  * 0xfde00000..0xfe000000  : consistent mem
19/11:10:19|  * 0xfde00000..0xfde00000  : early ioremap
19/11:10:19|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/11:10:19|NR_IRQS:512
19/11:10:19|clocksource: timebase mult[a00000] shift[22] registered
19/11:10:19|Console: colour dummy device 80x25
19/11:10:19|Mount-cache hash table entries: 512
19/11:10:19|NET: Registered protocol family 16
19/11:10:19|PCI: Probing PCI hardware
19/11:10:19|bio: create slab <bio-0> at 0
19/11:10:19|NET: Registered protocol family 2
19/11:10:19|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/11:10:19|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/11:10:19|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/11:10:19|TCP: Hash tables configured (established 4096 bind 4096)
19/11:10:20|TCP reno registered
19/11:10:20|NET: Registered protocol family 1
19/11:10:20|ROMFS MTD (C) 2007 Red Hat, Inc.
19/11:10:20|msgmni has been set to 214
19/11:10:20|io scheduler noop registered
19/11:10:20|io scheduler anticipatory registered
19/11:10:20|io scheduler deadline registered
19/11:10:20|io scheduler cfq registered (default)
19/11:10:20|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/11:10:20|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/11:10:20|console [ttyS0] enabled
19/11:10:20|brd: module loaded
19/11:10:20|loop: module loaded
19/11:10:20|Device Tree Probing 'ethernet'
19/11:10:20|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:b7
19/11:10:20|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/11:10:20|XLlTemac: DCR address: 0x80
19/11:10:20|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/11:10:20|XLlTemac: Allocating DMA descriptors with kmalloc
19/11:10:20|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/11:10:20|XTemac: PHY detected at address 7.
19/11:10:20|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/11:10:21|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21|Using buffer write method
19/11:10:21|cfi_cmdset_0001: Erase suspend on write enabled
19/11:10:21|cmdlinepart partition parsing not available
19/11:10:21|RedBoot partition parsing not available
19/11:10:21|Creating 5 MTD partitions on "fc000000.flash":
19/11:10:21|0x000000000000-0x000000500000 : "golden_firmware"
19/11:10:21|0x000000500000-0x000000800000 : "golden_kernel"
19/11:10:21|0x000000800000-0x000000d00000 : "user_firmware"
19/11:10:21|0x000000d00000-0x000000fe0000 : "user_kernel"
19/11:10:21|0x000000fe0000-0x000001000000 : "env_variables"
19/11:10:21|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/11:10:21|SPI: XIlinx spi: bus number now 32766
19/11:10:21|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/11:10:22|SPI: XIlinx spi: bus number now 32765
19/11:10:22|mice: PS/2 mouse device common for all mice
19/11:10:22|Device Tree Probing 'i2c'
19/11:10:22| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/11:10:22|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/11:10:22|TCP cubic registered
19/11:10:22|NET: Registered protocol family 17
19/11:10:22|RPC: Registered udp transport module.
19/11:10:22|RPC: Registered tcp transport module.
19/11:10:22|eth0: XLlTemac: Options: 0x3fa
19/11:10:22|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/11:10:23|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/11:10:23|eth0: XLlTemac: speed set to 1000Mb/s
19/11:10:25|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/11:10:25|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/11:10:25|Sending DHCP requests ., OK
19/11:10:26|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
19/11:10:26|IP-Config: Complete:
19/11:10:26|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
19/11:10:26|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/11:10:26|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
19/11:10:26|Looking up port of RPC 100003/2 on 192.168.11.99
19/11:10:26|Looking up port of RPC 100005/1 on 192.168.11.99
19/11:10:26|VFS: Mounted root (nfs filesystem) on device 0:12.
19/11:10:26|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/11:10:27|Starting sysinit...
19/11:10:27|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/11:10:27|		Press 'I' to enter interactive startup.
19/11:10:27|Setting clock  (utc): Thu Dec 19 10:10:28 GMT 2019 [  OK  ]
19/11:10:28|Building the cache [  OK  ]
19/11:10:28|Setting hostname aida04:  [  OK  ]
19/11:10:29|Mounting local filesystems:  [  OK  ]
19/11:10:30|Enabling /etc/fstab swaps:  [  OK  ]
19/11:10:32|Finishing sysinit...
INIT: Entering runlevel: 3
19/11:10:35|Entering non-interactive startup
19/11:10:36|FATAL: Module ipv6 not found.
19/11:10:37|Bringing up loopback interface:  [  OK  ]
19/11:10:39|FATAL: Module ipv6 not found.
19/11:10:39|Starting system logger: [  OK  ]
19/11:10:40|Starting kernel logger: [  OK  ]
19/11:10:40|Starting rpcbind: [  OK  ]
19/11:10:41|Mounting NFS filesystems:  [  OK  ]
19/11:10:42|Mounting other filesystems:  [  OK  ]
19/11:10:42|Starting xinetd: [  OK  ]
19/11:10:43|Starting midas:  Starting MIDAS Data Acquisition for aida04
19/11:10:43|xaida: device parameters: base=0x81000000 size=0x200000
19/11:10:48|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/11:10:49|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/11:10:49|xaida: driver assigned major number 254
19/11:10:49|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/11:10:54|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/11:10:54|AIDAMEM: aidamem: driver assigned major number 253
19/11:10:54|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/11:11:01|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/11:11:01|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/11:11:01|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/11:11:01|Computer Name = aida04; Temp Directory = /tmp/tcl361
19/11:11:05|package limit is not available: can't find package limit
19/11:11:07|Running with default file descriptor limit
19/11:11:07|package setuid is not available: can't find package setuid
19/11:11:09|Running as user 0 group 0
19/11:11:09|
19/11:11:10|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
19/11:11:10|
19/11:11:10|Built without pthreaxaida: open:
19/11:11:10|ds
19/11:11:10|
19/11:11:10|Creating NetAIDAMEM: aidamem_open:
19/11:11:10|Vars
19/11:11:10|Output buffer length = 65504; format option = 4; transfer option = 3
19/11:11:11|EB transfer option = 3
19/11:11:11|NetVars created and initialised
19/11:11:11|Statistics thread starting
19/11:11:11|Statistics thread created
19/11:11:11|Stat/Rate creation thread starting
19/11:11:11|Data Acquisition task has PID 375
19/11:11:11|Stat/Rate creation thread created
19/11:11:11|Hit/Rate creation thread starting
19/11:11:11|Hit/Rate creation thread created
19/11:11:11|AIDA Heartbeat thread starting
19/11:11:11|Heartbeat thread created
19/11:11:11|Installing signal handlers
19/11:11:11|Done
19/11:11:11|ModuleNum = 0
19/11:11:11|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 18430701 
19/11:11:11|Spectra table initialised
19/11:11:11|AIDA Data Acquisition now all ready to start
19/11:11:11|SIGBUS, SIGSEGV and SIGPIPE traps setup
19/11:11:11|/debug user "debug" password "-f9x7ruru8cg"
19/11:11:17|httpd started on port 8015
19/11:11:18|
19/11:11:18|Cannot use /MIDAS/config/TclHttpd/aida04@8015/startup.tcl
19/11:11:18|Custom startup from /MIDAS/config/TclHttpd/aida04/startup.tcl
19/11:11:18|XAIDA Access package 1.0
19/11:11:19|/XAIDAAccessServer
19/11:11:19|XAD9252 Access package 1.0
19/11:11:20|/XAD9252AccessServer
19/11:11:20|/DataBaseAccessServer
19/11:11:20|/NetVarService
19/11:11:20|/SigTaskService
19/11:11:20|Loaded MemSasAccess
19/11:11:20|/SpectrumService
19/11:11:20|loading tcl/AIDARunControl.tcl for namespace ::
19/11:11:20|/DataAcquisitionControlServer
19/11:11:20|DefineMessage unknown
19/11:11:20|Run Control Server Implementation for AIDA
19/11:11:21|RunControlServer loaded
19/11:11:21|loading Html/RunControl/implementation.tcl
19/11:11:21|[  OK  ]
19/11:11:21|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
19/11:11:21|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
19/11:11:21|
19/11:11:21|DENX ELDK version 4.2 build 2008-04-01
19/11:11:21|Linux 2.6.31 on a ppc
19/11:11:21|
19/11:11:21|aida04 login: Created UI registers
19/11:11:22|RunControl loaded
19/11:11:22|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
19/11:11:22|AIDA RunControl loaded
19/11:11:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
Attachment 1: 300.png
300.png
  110   Thu Dec 19 09:22:52 2019 TDThusrday 19 December
10.15 Found DAQ running at c. 1M data items/s 

      Rate spectra show aida09 1.*.H and one 1x LEC channel of aida08 asic running hot

      Check ASIC settings *all* FEE64s *all* ASICs restores rates to normal


      /TapeData/171219/R2_* files indicates files R2_0 ... R2_1 are probably OK and
      problem started c. 08.00 - activity in S4?

-rw-rw-r--. 1 npg npg 2097152000 Dec 19 01:56 R2_0
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 05:30 R2_1
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:05 R2_2
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:15 R2_3
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:29 R2_4
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:34 R2_5
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:38 R2_6
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:43 R2_7
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:47 R2_8
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:52 R2_9
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:56 R2_10
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:00 R2_11
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:05 R2_12
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:09 R2_13
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:14 R2_14
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:18 R2_15
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:23 R2_16
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:27 R2_17
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:31 R2_18
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:36 R2_19
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:40 R2_20
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:45 R2_21
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:49 R2_22
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:54 R2_23
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:58 R2_24
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:02 R2_25
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:07 R2_26
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:11 R2_27
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:16 R2_28
-rw-rw-r--. 1 npg npg 1582563328 Dec 19 10:19 R2_29

10.31 1.8.L and Stat spectra - attachments 1-3

10.32 DAQ stop file 171219/R2_30

      zero *all* histograms

      Check ASIC settings *all* FEE64s *all* ASICs


10.48 system wide checks OK except aida07 calibration & master clock status fail - attachments 4-9

      GSI WR timestamp control OK - attachments 10-11

      DSSSD biases & leakage currents OK - attachment 12
      FEE64 temperatures OK - attachment 13
      Good event statistics OK - attachment 14

11.31 DAQ/Merger/TapeServer stopped merging/writing data at start of new run

      Stop DAQ/Merger/TapeServer, reboot FEE64s and restart from scratch

      DAQ start file 171219/R8

      slow comparator 0x64

      alpha background

      data rate c. 170kb/s

15.30 DAQ continues OK file 171219/R9_0
    
      DSSSD biases & leakage currents OK - attachment 15
      BNC Pb-5 settings - attachment 16
      FEE64 temperatures OK - attachment 17
      Good event statistics OK - attachment 18

      system wide checks OK except master clock status fail
  
      GSI WR timestamp control OK - attachments 19-20

      Merger & server consoles - attachment 21
Attachment 1: 100.png
100.png
Attachment 2: 101.png
101.png
Attachment 3: 102.png
102.png
Attachment 4: 110.png
110.png
Attachment 5: 111.png
111.png
Attachment 6: 112.png
112.png
Attachment 7: 113.png
113.png
Attachment 8: 114.png
114.png
Attachment 9: 115.png
115.png
Attachment 10: 116.png
116.png
Attachment 11: 117.png
117.png
Attachment 12: 200.png
200.png
Attachment 13: 201.png
201.png
Attachment 14: 202.png
202.png
Attachment 15: 1.png
1.png
Attachment 16: 2.png
2.png
Attachment 17: 3.png
3.png
Attachment 18: 4.png
4.png
Attachment 19: 5.png
5.png
Attachment 20: 6.png
6.png
Attachment 21: 8.png
8.png
  109   Wed Dec 18 17:39:27 2019 TDWednesday 18 December
18.40 Detector biases & leakage currents OK - attachment 1
      FEE64 temperatures OK - attachment 2
      FEE64 good event statistics OK - attachment 3
      3x <20k, 8x <50k, 11x <100k
      note - all 12x FEE64s zero disc-info #6

      system wide checks - see attachments 4-9
      master clock status fail
      WR decoder status - aida08 fails

18.49 GSI WR status control & timestamps - attachments 10 & 11

18.50 check ASIC control *all* FEE64s *all* ASICs

18.58 Rate, Stat, 1.8.L and 1.8.H spectra all FEE64s - attachments 12-17
      OK except aida10 no HEC data & low amplitude LEC data

19.00 *all* histograms zero'd

19.03 NewMerger, TapeServer and full screen - attachments 18-20

19.07 Options, ASIC settings aida09 & aida10 - attachments 21-23

20.15 collect all WR status erro counters for baseline
      WR decoder status - no errors

21.56 FRS DAQ crashed and not currently recoverable
      Stop current run and run background alphas overnight

21.57 run stopped file 171219/R1_304

      slow comparator threshold 0xf -> 0x64
      check ASIC control *all* FEE64s *all* ASICs

      BNC PB-5 attenuation x1 -> x1000

      *all* histograms zero'd

22.02 DAQ start file 171219/R2

      background alphas



 
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 11.png
11.png
Attachment 12: 21.png
21.png
Attachment 13: 22.png
22.png
Attachment 14: 23.png
23.png
Attachment 15: 24.png
24.png
Attachment 16: 25.png
25.png
Attachment 17: 26.png
26.png
Attachment 18: 30.png
30.png
Attachment 19: 31.png
31.png
Attachment 20: 32.png
32.png
Attachment 21: 40.png
40.png
Attachment 22: 41.png
41.png
Attachment 23: 42.png
42.png
  108   Wed Dec 18 02:44:27 2019 NH17.12.19, 18.12.19
Beam: 238U (850 MeV/u)
Fragment: 168Re

Tuning accelerator and FRS, believed since 3AM we are implanting on AIDA.

FEE issues:
One HEC in aida03 seems to be empty - LEC channel fills
aida09 has a lot of weird channels, check/load doesn't seem to fix
aida10 still no data after cable replaced

DSSD leakages OK - slightly higher now

Temps good


Fig 4: Rates 14:32

Leakages and temps good AS Before

Checking if we are implanting properly - possibly into the first DSSD at the moment

Fig 5 FRS PID
Fig 6 DSSD 
Fig 7 DSSD (PID gated)
Attachment 1: 1812-stats.png
1812-stats.png
Attachment 2: 1812-leakages.png
1812-leakages.png
Attachment 3: 1812-temps.png
1812-temps.png
Attachment 4: 1812v2-Rates.png
1812v2-Rates.png
Attachment 5: nh_1712_zaoq.png
nh_1712_zaoq.png
Attachment 6: nh_1712_dssd.png
nh_1712_dssd.png
Attachment 7: nh_1712_dssd_id.png
nh_1712_dssd_id.png
  107   Mon Dec 16 14:48:02 2019 NH TDAIDA 16.12.19
DSSD Snout was reassembled to install the second bPlast (downstream) 
 (fig 1)

Ribbon cable for aida10 was replaced (DSSD3 upper)

All DSSDs bias again (see fig 2)

Alpha run in progress - channels OK with 2 exceptions  (fig 3)

- aida10 still recording minimal to no counts

- channel 55 aida08 reading lots of data even with comparator 0x64 - corresponds to DSSD1 strip 0 (bottom strip)
  possible ground/cable issue? will check when access possible (fig 4)

Unusual MIDAS error noticed (fig 5) - restart of HTTPd fixed it
Attachment 1: 20191213_103710.jpg
20191213_103710.jpg
Attachment 2: 161219_aida_DSSDs.png
161219_aida_DSSDs.png
Attachment 3: 161219_rates.png
161219_rates.png
Attachment 4: 161219_aida0855.png
161219_aida0855.png
Attachment 5: midasmessage131219.png
midasmessage131219.png
  106   Wed Dec 11 10:58:52 2019 NHInitital Dec. Analysis
Fig 1: 2D hit pattern of each AIDA DSSD for MBS run 24 (40Ar primary beam, centred)

Low statistics in DSSD2 partly due to low energy deposition 

Fig 2: Position according to TPC42 (ca. 2 metres before AIDA)

Fig 3: 2D hit pattern of DSSD 1 for MBS run 37 (34Si fragments, no PID gate)

-
Bottom part of AIDA noticably cut off - rotation in DSSD also noticed (not an issue)

Snout was investigated and platform was discovered to not be fully level by Bogdan using a laser level - he has adjusted it so that the AIDA snout is now level to within 1 mm.
(The beam right side was previous 4-6mm below the beam left side)

Y cut in fragment run is currently of unknown origin. Optics are being investigated

-

Fig 4: Hit pattern of 40Ar with strip offsets fixed (no missing strips)

Fig 5: Energy loss in DSSD1 of 40Ar primary beam

Fig 6: TPC 41, TPC 42, AIDA X comparison

Fig 7: " Y comparison

Fig 8-11: As above for 34Si run

Y cut unknown still.


 
Statistics of 40Ar file
======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Fri Dec  6 05:21:42 2019
Last event : Fri Dec  6 05:26:17 2019
Duration   : 274.604 seconds
--------------------------------------
MBS Events   (M): 131808426 (481 kHz)
AIDA Words   (W): 110608065 (403 kHz)
ADC Words    (A): 36869349 (134 kHz)
ADC Decays   (D): 36782621 (134 kHz)
ADC Implants (I):    86728 (316 Hz)
Info Words   (F): 73738716 (269 kHz)
SYNC48 Words (S): 36869354 (134 kHz)
SYNC63 Words (s): 36869354 (134 kHz)
PAUSE Words  (P):        4 (0 Hz)
RESUME Words (R):        4 (0 Hz)
Discrim Words(D):        0 (0 Hz)
Unknown Words(U):        0 (0 Hz)
Timewarps    (T):        0 (0 Hz)
Dead Time    (X): 0.113631 s (0.0413799%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |        W        A        D        I        F        S        s    P    R    D    U    T        X
   1 |  4383018  1461006  1439401    21605  2922012  1461006  1461006    0    0    0    0    0 0
   2 | 11468351  3822783  3814577     8206  7645568  3822783  3822783    1    1    0    0    0 0.00733666
   3 |  4353624  1451208  1428500    22708  2902416  1451208  1451208    0    0    0    0    0 0
   4 |  8355002  2785000  2755870    29130  5570002  2785001  2785001    0    0    0    0    0 0
   5 |  1133544   377848   376489     1359   755696   377848   377848    0    0    0    0    0 0
   6 |  9617940  3205980  3205767      213  6411960  3205980  3205980    0    0    0    0    0 0
   7 |  4385477  1461825  1460962      863  2923652  1461826  1461826    0    0    0    0    0 0
   8 | 15105135  5035043  5032985     2058 10070092  5035045  5035045    1    1    0    0    0 0.0126429
   9 | 37487211 12495735 12495422      313 24991476 12495736 12495736    2    2    0    0    0 0.0936512
  10 |   110316    36772    36772        0    73544    36772    36772    0    0    0    0    0 0
  11 | 12329904  4109968  4109763      205  8219936  4109968  4109968    0    0    0    0    0 0
  12 |  1878543   626181   626113       68  1252362   626181   626181    0    0    0    0    0 0


Statistics of 34Si file
======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Sat Dec  7 05:20:22 2019
Last event : Sat Dec  7 05:23:51 2019
Duration   : 208.635 seconds
--------------------------------------
MBS Events   (M): 148639738 (714 kHz)
AIDA Words   (W): 132033086 (634 kHz)
ADC Words    (A): 44011024 (211 kHz)
ADC Decays   (D): 43874168 (210 kHz)
ADC Implants (I):   136856 (657 Hz)
Info Words   (F): 88022062 (423 kHz)
SYNC48 Words (S): 44011028 (211 kHz)
SYNC63 Words (s): 44011028 (211 kHz)
PAUSE Words  (P):        3 (0 Hz)
RESUME Words (R):        3 (0 Hz)
Discrim Words(D):        0 (0 Hz)
Unknown Words(U):        0 (0 Hz)
Timewarps    (T):        0 (0 Hz)
Dead Time    (X): 0.0806864 s (0.0386734%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |        W        A        D        I        F        S        s    P    R    D    U    T        X
   1 |  6814953  2271651  2255653    15998  4543302  2271651  2271651    0    0    0    0    0 0
   2 | 14012582  4670860  4668280     2580  9341722  4670860  4670860    1    1    0    0    0 0.0088226
   3 |  7618290  2539430  2523614    15816  5078860  2539430  2539430    0    0    0    0    0 0
   4 |  8928934  2976310  2950380    25930  5952624  2976312  2976312    0    0    0    0    0 0
   5 |  1631415   543805   531957    11848  1087610   543805   543805    0    0    0    0    0 0
   6 | 10908021  3636007  3633830     2177  7272014  3636007  3636007    0    0    0    0    0 0
   7 |  5644296  1881432  1864053    17379  3762864  1881432  1881432    0    0    0    0    0 0
   8 | 14863755  4954585  4931620    22965  9909170  4954585  4954585    0    0    0    0    0 0
   9 | 36745143 12248379 12244135     4244 24496764 12248380 12248380    2    2    0    0    0 0.0718638
  10 |   198573    66191    66191        0   132382    66191    66191    0    0    0    0    0 0
  11 | 15101900  5033966  5018830    15136 10067934  5033967  5033967    0    0    0    0    0 0
  12 |  9565224  3188408  3185625     2783  6376816  3188408  3188408    0    0    0    0    0 0

All good (no deadtime issues)
Attachment 1: AIDA_HitDec19.png
AIDA_HitDec19.png
Attachment 2: TPCs.png
TPCs.png
Attachment 3: AIDA-fragments.png
AIDA-fragments.png
Attachment 4: AIDA_XY.png
AIDA_XY.png
Attachment 5: AIDA_E.png
AIDA_E.png
Attachment 6: TPC_41_42_AIDA_X.png
TPC_41_42_AIDA_X.png
Attachment 7: TPC_41_42_AIDA_Y.png
TPC_41_42_AIDA_Y.png
Attachment 8: AIDA_XY_34Si.png
AIDA_XY_34Si.png
Attachment 9: AIDA_E_34Si.png
AIDA_E_34Si.png
Attachment 10: TPC_41_42_AIDA_X_34Si.png
TPC_41_42_AIDA_X_34Si.png
Attachment 11: TPC_41_42_AIDA_Y_34Si.png
TPC_41_42_AIDA_Y_34Si.png
  105   Sat Dec 7 02:39:02 2019 DK, NH et al.34Si Fragments PID
See attached

Tentative PID (DK) as attachment #3

DESPEC LISE++ file as attachment #4
Attachment 1: 2019-12-07-PID.png
2019-12-07-PID.png
Attachment 2: 59_AM.png
59_AM.png
Attachment 3: pid.png
pid.png
Attachment 4: Ar-300_S4_updated_matter_DEPSEC_34Si.lpp
Version 11.0.88

{============================= Main Part ======================================}
[general]
                    File = C:\Users\Nic\Documents\Ar-300_S4_updated matter_DEPSEC 34Si.lpp
                    Date = 06-12-2019
                    Time = 23:05:23
           Configuration = GSI\FRS-TA2-S4_2014.lcn
             Optionsfile = GSI_FRS_2012.lopt
                   Title = GSI FRS - TA2 -2014
          BlockStructure = MMMDSWMDMSAMMMMMMWMMDMSDMMMMMMMMMMMMMMMMMMSMMMWMMMMMMMMMMMMMMMMMM

[settings]
                   A,Z,Q = 40Ar18+                               ; Mass ElementName Charge+ Beam
                  Energy = 300                  MeV/u            
               Intensity = 1e+7                 pps              ; enA,pna,pps,kW
            RF frequency = 20                   MHz              
            Bunch length = 1                    ns               
         Settings on A,Z = 34Si                                  ; Mass ElementName Charge+ Beam

[OpticsBeam]
                      BX = 1                    (±)mm            ; one-half the horisontal beam extent (x)
                      BT = 5                    (±)mrad          ; one-half the horisontal beam divergence(x')
                      BY = 1.5                  (±)mm            ; one-half the vertical beam extent (y)
                      BF = 3.33                 (±)mrad          ; one-half the vertical beam divergence (y')
                      BL = 0                    (±)mm            
                      BD = 0.05                 (±)%             ; one-half of the momentum spread (dp/p)
                  ShiftX = 0                    mm               ; beam respect to the spectrometer axis
                  AngleX = 0                    mrad             ; beam respect to the spectrometer axis
                  ShiftY = 0                    mm               ; beam respect to the spectrometer axis
                  AngleY = 0                    mrad             ; beam respect to the spectrometer axis
            Scheme Angle = 0                    degrees          
                  ShapeX = 1
                  ShapeT = 1
                  ShapeY = 1
                  ShapeF = 1
                  ShapeL = 1
                  ShapeD = 1
               OptBeam_X = 1                    (±)mm            
               OptBeam_T = 30                   (±)mrad          
               OptBeam_Y = 1                    (±)mm            
               OptBeam_F = 30                   (±)mrad          
               OptBeam_L = 0                    (±)mm            
               OptBeam_D = 1.5                  (±)%             

[options]
               NP simple = 32                                    ; Number of points in distribution
        NP charge states = 32                                    ; Number of points in distribution
                NP wedge = 32                                    ; Number of points in distribution
           Charge states = No                                    ; No  & Yes
           CutEdgeEffect = 1                                     ; 1-Yes. Default, 0-no - for extended configurations
       Prim.beam scatter = 0                                     ; 0-without, 1-with
              Delta peak = 0                                     ; 0-without, 1-with
             BrhoMeanMax = 1                                     ; 0-Max, 1-Mean
            BrhoMeMaLeRi = 3                                     ; 0-Max, 1-Mean, 2-Left, 3-Right /for fission/

[target]
         Target contents = 0,4,1,9.012                           ; Nomer,Z,Atoms,Mass
        Target thickness = 1,2500.7,1.848,0,0,0                  ; State,Thickness,density,angle,SolidGas,..
  Target fusion compound = 0
   Targ use for Q-states = 1
           Target Defect = 1,0                                   ; [0] choice - % or micron at 0 degree,  [1]=value; 
       Degrader contents = 0,22,1,47.9                           ; Nomer,Z,Atoms,Mass
      Degrader thickness = 0,35,4.519,0,0,0                      ; State,Thickness,density,angle,SolidGas,..
  Degra use for Q-states = 1
         Degrader Defect = 1,0                                   ; [0] choice - % or micron at 0 degree,  [1]=value; 

[mechanism]
                Reaction = 0                                     ; 0 - fragm, 1 - fusion-resid, 2 - fusion-fission
               CalcOther = 1                                     ; calculate other reactions
           V calculation = 5                                     ; 0 - constant, 1 - Borrel, 2 - Rami, 3-convolution, 4-two body reaction
             V_opt/Vbeam = 1                                     ; default 1
         Velocity_exceed = 1                                     ; 0 - without, 1-with - two-body recations velocity corrections
Binding Energy for Vf/Vp = 8                    MeV              ; Binding energy for Borrel's expression
    Shift for Vf/Vp calc = 0     
        Prefragment_Rami = 1                                     ; 1-Yes, 0-No
                  Sigma0 = 90                   MeV/c            ; default 90
                  SigmaD = 0                    MeV/c            ; default 200
                  SigmaM = 87                   MeV/c            ; default 87
               Asymmetry = 0                    %                ; default 0
          Method v-sigma = 0                                     ; 0 - Goldhaber, 1-Morrissey,2-Friedman,3-Convolution
               G_Surface = 0.95                 MeV/fm^2         
 Symmetry around half_Ab = 1                                     ; 1 - yes, 0-no
 Pfaff pickup correction = 0                                     ; 1 - yes, 0-no
     ChargeExchangePfaff = 0                                     ; 1 - exclude, 0-forget
            Sigma corr 0 = 0                                     ; Coulomb energy
            Sigma corr 1 = 0                                     ; Projectile mass
           Friedman mode = 2                                     ; 0-Qgg, 1-Surface, 2-Qgg+Surface
         Prefragment_Fri = 1                                     ; 1-Yes, 0-No
        Coulomb_Friedman = 1                                     ; 1-Yes, 0-No
                  K_Morr = 8                    MeV/A            ; E/A=8MeV/A default; D.Morrissey coef.
               K_MorHalf = 8                    MeV/A            ; E/A=8MeV/A default at Afrag=Aproj/2; D.Morrissey coef.
                 AA_fast = 0                                     ; 1-Yes, 0-No
            BarrierShape = 1                                     ; 0-classical, 1-quantum mech.
                 H_omega = 5                    MeV              ; default 3
           Probabilty_CN = 0                                     ; 0/1 use Prbabilty for CN formation
            UseVanishing = 1                                     ; 1-Yes, 0-No
              VanishMode = 0                                     ; 0-Sierk, 1-Cohen
           NuclPotential = 1                                     ; 0-Bass, 1-WS
                   WS_V0 = 105                  MeV              
                   WS_R0 = 1.12                 fm               
                    WS_a = 0.75                 fm               
          FusDiffuseness = 1    
              Width Coef = 1                                     ; default 1; for Leon's charge state distribution
          gZt Correction = 1                                     ; default 1;      Leon's C.S.D.
           PowerCoefLeon = 0.477                                 ; default 0.477; Leon's C.S.D.
           Cross section = File                                  ; Fit  & File
           Charge method = 3                                     ; charge calculations method  0-5
      EPAX Cross Section = 4                                     ; cross section calculations method 0-4
        SR Cross Section = 2                                     ; EPAX for SR  0-2
             Energy Loss = 2                                     ; energy loss calculation method 0-3
     Anglular straggling = 1                                     ; 0-LISE, 1-ATIMA
         StragglingCoef1 = 0.217
         StragglingCoef2 = 1.12 
       Energy straggling = 1                                     ; 0-LISE, 1-ATIMA
       EnergyStragMethod = 0                                     ; 0-integrate, 1-table
        EnergyStragShape = 0                                     ; 0-Gauss, 1-Landau-Vavilov
          EquilThickness = 1                                     ; 0-Charge, 1-Global
              MassMethod = 0                                     ; 0-DB+calcul, 1 + just calcul
            MassDataBase = 0                                     ; 0-A&W, 1-User ME
            Mass formula = 2                                     ; 0-LDM, 1-Myerer, 2: 1+corrections
      UseChargeForEnergy = 2                                     ; 0-No, 1-Yes, 2-Auto
         EnergyValueAuto = 30                                    ; default value 30 MeV/u
         EquilibriumMode = 1                                     ; 0-Equil, 1-NonEquil
               UB_Global = 70                                    ; default 70 MeV/u
             MinZ_Global = 2                                     ; default Z>=29
        ChargeStateOptim = 1                                     ; 0-No, 1-Yes
         ZmQ_AfterReactn = 0                                     ; default 0 (full stripped)
             EPAX_p_Norm = 1
               EPAX_p_Un = 1.65
              EPAX_p_Up0 = 1.79
              EPAX_p_Up1 = 0.00472
              EPAX_p_Up2 = -1.3e-5 
                EPAX_p_H = 1

[fission]
         FisAngDistShape = 0                                     ; 0-isotropic; 1-anisotropic
     FisMomCutForAngDist = 2                                     ; 0-dont use;  1-use just MatrixKinematics; 2-use for all; (default 2)
      OddEvenCorrections = 1                                     ; 0-dont use;  1-use
 PostScissionEvaporation = 1                                     ; 0-dont use;  1-use
   DeexcitFunctionPoints = 0                                     ; 0- average deexcitation energy;  1- 3 points; 2 - manually
           FisEXmanually = 20                                    ; Excitation energy manually
           FisCSmanually = 1000                                  ; Cross section manually
            FisTXEmethod = 1                                     ; 0-from Edissipated, 1 from Q-value
                   Fis_f = 0.0045                                ; default 0.0045
              FisEXsigma = 5.5                  MeV              ; default 5.5
            FisCS_Global = 1e-11    
               FisCS_TKE = 1e-8     
                FisBeta1 = 0.625                                 ; deformation of light fragment
                FisBeta2 = 0.625                                 ; deformation of heavy fragment
                FisTKE_d = 2                    fm               ; d-param in Wilkins formula
              FisBetaFit = 1                                     ; 0-manual, 1-fit
                      N0 = 83                                    ; default 82
                     dU0 = -2.65                                 ; default -2.5
                      C0 = 0.7                                   ; default 1.4
                   cpol0 = 0.65                                  ; default 0.65
                  width0 = 0.63                                  ; default 0.63
                      N1 = 90                                    ; default 90
                     dU1 = -3.8                                  ; default -5.5
                      C1 = 0.15                                  ; default 0.16
                   cpol1 = 0.55                                  ; default 0.55
                  width1 = 0.97                                  ; default 0.97

[charge_suppression]
                 FragInd = 1e-3     
               FragTotal = 1e-5     
                 BeamInd = 1e-20    
               BeamTotal = 1e-20    

[convolution]
        Convolution mode = 1                                     ; 0-Qgg, 1-Surface, 2-Qgg+Surface
               SigmaConv = 91.5                 MeV/c            ; default 90 for Convolution
              CoefConv_0 = 3.344 
              CoefConv_1 = 3     
              CoefConv_2 = 2.936 
             ShiftConv_0 = 0.158 
             ShiftConv_1 = 0.149 
             ShiftConv_2 = 0.153 

[evaporation]
          NP evaporation = 32                                    ; Number of points in distribution
              EvapMethod = 2
        StateDensityMode = 2                                     ; 0, 1+pairing, 2+shell
      EvapUnstableNuclei = 1                                     ; 0 - only stable,1 +unstable
              Tunnelling = 1                                     ; 1-Yes, 0-No
          AvoidUnboundCS = 1                                     ; 1-Yes, 0-No
       ProtectedChannels = 1                                     ; 1-Yes, 0-No
           R_Evaporation = 5.7                  fm               ; correction for the effective Coulomb barrier
         Mode_Apf_manual = 0                                     ; 1-manual, 0-auto
             Energy_in_T = 2                                     ; default 2
     EvaporationVelocity = 0                                     ; 0 - quality, 1 -fast
        DeltaOddEvenEvap = 12    
     DeltaOddEvenFission = 14    
   BreakupTemperature250 = 4.7  
   BreakupTemperature150 = 5.9  
   BreakupTemperature050 = 8    
      BreakupDiffuseness = 0.05 
      DissipationKramers = 0                                     ; 0 - no, 1 - use
 DissipationStepFunction = 1                                     ; 0 - no, 1 - use
         DissipationBeta = 1                                     ; default 2.0
                 mode_1n = 1                                     ; 1-Yes, 0-No
                 mode_2n = 0                                     ; 1-Yes, 0-No
                 mode_1p = 1                                     ; 1-Yes, 0-No
                 mode_2p = 0                                     ; 1-Yes, 0-No
                  mode_a = 1                                     ; 1-Yes, 0-No
                  mode_d = 0                                     ; 1-Yes, 0-No
                  mode_t = 0                                     ; 1-Yes, 0-No
                mode_3he = 0                                     ; 1-Yes, 0-No
                mode_fis = 1                                     ; 1-Yes, 0-No
             mode_brk_up = 1                                     ; 1-Yes, 0-No
              mode_gamma = 0                                     ; 1-Yes, 0-No

[fission_barrier]
    FissionBarrierFactor = 1    
      FissionBarrierMode = 1                                     ; #0-4
      OddEvenCorrections = 1                                     ; 1-Yes, 0-No
        ShellCorrections = 1                                     ; 1-Yes, 0-No
             FB_InOutMax = 2                                     ; #0-2 - in/out/max
             ModeForUser = 1                                     ; #0-2
           NdeltaOddEven = 2.5  
           ZdeltaOddEven = 9    

[excitation_energy]
       GeomAA_Correction = 1                                     ; 0 - don't use,1 - use -default
          Thermalization = 0                                     ; 1-Yes, 0-No
          ThermaTimeCoef = 3e+0                                  ; 2.1e-22 MeV *s/e(t)
                Friction = 0                                     ; 0 - off,1 - on
          Ev_A_SigmaCoef = 9.6  
         G_FrictionCoef1 = 6.5  
         G_FrictionCoef2 = 0    
           G_FactorCoef1 = 1.5  
           G_FactorCoef2 = 2.5  
               DepthHole = 40    
          EnergyCoef_CB0 = 0       
          EnergyCoef_CB1 = 13.3     
          EnergyCoef_CB2 = 0       
           SigmaCoef_CB0 = 0       
           SigmaCoef_CB1 = 9.6     
           SigmaCoef_CB2 = 0       
              D_MeanTemp = 13       
               AA_factor = 1       
          ApplyLimitTemp = 0                                     ; 1-Yes, 0-No

[evapauto]
                  tun_a0 = -0.61392
                  tun_a1 = 0.44559
                  tun_a2 = 0.12008
                 A_Bound = 300                                   ; mass
               A_Pairing =  70                                   ; mass

[plot]
            Start target = Detector                              ; Detector & RF      
            Start of TOF = M7
            Stop  of TOF = M34
         dE-detector-1st = M28
         dE-detector-2nd = M7
            TKE-detector = M41
                PlotBrho = D1
               PlotWedge = S4
              X-detector = M19
              Y-detector = M19
                 Tilting = M1
                 Stopper = W1
                RO_Wedge = n
          ConditionBlock = A10
          Plot threshold = 1e-10                pps              ; minimal value for plot scale
     Shift of TOF for RF = 0                    ns               ; for dE-TOF plot with RF
  Fraction of RF trigger = 2
            UseCondition = 0
         TKE_calibration = 1,1,0,MeV                             ; Input PV(0) or CH(1), A, B, dimension

[cs_file]
              UserDiffCS = 0                                     ; Number of User Diff CS saved in this file
         AppendOverwrite = 1
          AttachedInside = 1
            ShowCSinPlot = 1
                    Chi2 = 1
              CSfilename = 

[sec_reactions]
        NP sec.reactions = 16                                    ; Number of points in distribution
     Secondary reactions = 0                                     ; 0/1 - use secondary reactions in calculations
         fiss_FilterUse0 = 1
         fiss_FilterUse1 = 1
         fiss_FilterUse2 = 1
            fiss_ellipse = 5
          fiss_NdeltaTop = 0
          fiss_ZdeltaTop = 0
          fiss_NdeltaBot = 25
          fiss_ZdeltaBot = 20
         frag_FilterUse0 = 3
         frag_FilterUse1 = 3
         frag_FilterUse2 = 3
            frag_ellipse = 4
          frag_NdeltaTop = 5
          frag_ZdeltaTop = 5
          frag_NdeltaBot = 6
          frag_ZdeltaBot = 6

[abrasion_fission]
... 2087 more lines ...
  104   Fri Dec 6 20:34:56 2019 CA, DK, NHDec 6 Night Shift
21.35 AIDA DAQ currently running, TapeService on, not writing to storage

      System wide checks ok

      Attachments 1-3: temperatures, good event stats, detector bias/leakage currents ok

22.04 waiting for beam. Possibility of fragments, further news to come

22.43 expecting fragments with 300 MeV/u 40Ar primary beam

      histograms zeroed

00.31 still waiting for beam

      detector bias/leakage currents ok (see google spreadsheet)

      FEE64 temps and good event statistics ok

02:02 still awaiting beam - issues with interlock

03:17 beam tuning - centering primary beam on target

03:43 beam reached AIDA, fragmentation beam (see elog entry by DK for PID)

      primary beam 300 Mev/u 40Ar, fragments in region of 34Si

      started writing to file Dec19/R2 at rate ~ 4000 kb/s

      histograms zeroed

03.49 system wide checks all ok

      attachments 4-6:

      temperatures ok

      bias/leakage currents ok

      good event statistics ok, except aida09 running slightly fast

04:03 attachments 7-8 - 1.8.H energy spectra

                      - no high energy events seen in aida10 -> checked ASIC control

                      - still no high energy events in aida10

04:09 attachment 9 - rate spectra for all FEE modules

04.16 attachment 10 - 1D hits for all FEE modules

04.19 beam stop - file R2_3

04:52 beam back on - file R2_6 - rate of ~ 5000 ions per spill

06:13 temperatures ok

      good event statistics ok (aida09 still running fast) - attachment 11

      detector bias/ leakage currents ok - see google spreadsheet

      all system wide checks ok

      1.8.H spectra - attachments 12/13

06:20 attachment 14 - rate spectra for all FEE modules

      attachment 15 - 1D hits for all FEE modules

      still no high energy events in aida10


      
Attachment 1: 191206_2139_temp.png
191206_2139_temp.png
Attachment 2: 191206_2140_bias.png
191206_2140_bias.png
Attachment 3: 191206_2140_stats.png
191206_2140_stats.png
Attachment 4: 191107_0351_temp.png
191107_0351_temp.png
Attachment 5: 191207_0352_bias.png
191207_0352_bias.png
Attachment 6: 191207_0352_stats.png
191207_0352_stats.png
Attachment 7: 191207_18h_2.png
191207_18h_2.png
Attachment 8: 191207_18H_1.png
191207_18H_1.png
Attachment 9: 191207_0408_rates.png
191207_0408_rates.png
Attachment 10: 191207_0415_statspec.png
191207_0415_statspec.png
Attachment 11: 191207_0614_stats.png
191207_0614_stats.png
Attachment 12: 191207_0606_18H_1.png
191207_0606_18H_1.png
Attachment 13: 191207_0609_18h_2.png
191207_0609_18h_2.png
Attachment 14: 191207_0618_rate.png
191207_0618_rate.png
Attachment 15: 191207_0619_hits.png
191207_0619_hits.png
  103   Thu Dec 5 22:00:11 2019 CA, DK, NHDec 5 Night Shift
15:30
  TD found the merger was running at ~43 Mb/s
  The odd-numbered FEE64s all had significant (>200k) disk rates
  ALL LEC/MEC fast discriminators and increased slow comparator from 0xa to 0x14
  Merger rate dropped to a more nominal value like 2.3 Mb/s
  Settings not saved

40Ar beam 300 MeV/u expected, but SIS problem at the moment.

23:00
  Leak currents are stable as previously (20-21 Nov) and entered to Google spreadsheet
  S4 vault is closed.
  Presently AIDA is running without tape storage
  System wide checks: 
    Master clock / aida01 sync reports error as expected; 
    All white rabbit decoders fail, but they are just off by 1 bit (no problem)
    Everything else okay.
  Stats look okay, but keep in mind above discriminator threshold -- see attachment 1
  Temperatures fine -- see attachment 2
  MBS looks okay -- we see a possible hint for the 1 bit difference with an "unexpected data word" -- see attachment 3 
  Data rate is 1.7 Mb/s, merger okay -- see attachment 4

23:30 
  Now we wait for the beam

00:05
  Attachments 5-11: results of system wide checks after collecting sync errors and WR errors from baseline - all ok

00:25
  Histograms zeroed at 00:25.

00:35

  1.8.L pulser peak widths

FEE64       width(ch)
  1            92
  2            131
  3            75
  4            97
  5            89
  6            117
  7            90
  8            87
  9            223
 10            58
 11            112
 12            126 

no peak observed on aida09

For comparison, pulser peaks measured on 13/11/19:

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

2:00
  40Ar beam tuning is finished.  Intensity at S4 is about 3 kHz per spill.  1 spill per / 10 seconds
  MBS system cannot be used for online merging tonight, because NeuLAND group left their connection open
  We can merge offline using the White Rabbit timestamps
  Remove the beam blocking steel flange from in front of AIDA et al.
  Writing to file Dec19/R1

03.20 Rate, Stat, 1.8.L, 1.8.H, 1.8.W spectra - attachments 13-20
      Observe
       light ions in 1.8.L spectra
       low energy (<1.5GeV) heavy ions in 1.8.H spectra with most heavy ions stopping in DSSSDs 1 & 2
       Stat hit patterns indicate broad (c. 5cm FWHM) centred heavy ion peak

03.24 Detector biases & leakage currents OK - attachment 21
      FEE64 temperatures OK - attachment 22
      Good event statistics (12 of 12 < 50k) - attachment 23

      System wide checks
      List All Sync Error Counters - attachment 24
      List all Sync Received Counters - attachment 25

      Baseline invoked earlier c. 00.05
      Collect all SYNC Error Counters for Baseline
      Collect All White Rabbit status Error Counters for Baseline 

04:07 2.6 mm Al degrader. 300Hz beam. 300 MeV/u 40Ar beam.

04.14 pulser peak seen in aida09 after performing ASIC check-load earlier in shift - width = 219 ch

04.45 1.8.H energy spectra (attachment 26/27)

06:08 attachments 28-30: detector bias/leakage currents ok
                         temperatures ok
                         good event statistics ok
                         all system wide checks ok

08:49 DAQ stop/start. TapeServer writing to No Storage. End of beam.


      
Attachment 1: 05-Dec-Stats.png
05-Dec-Stats.png
Attachment 2: 05-Dec-Temps.png
05-Dec-Temps.png
Attachment 3: 05-Dec-MBS.png
05-Dec-MBS.png
Attachment 4: 05-Dec-MergerRates.png
05-Dec-MergerRates.png
Attachment 5: adccalib.png
adccalib.png
Attachment 6: clockstatus.png
clockstatus.png
Attachment 7: meminfo.png
meminfo.png
Attachment 8: syncerrors.png
syncerrors.png
Attachment 9: syncpulses.png
syncpulses.png
Attachment 10: wrdecoder.png
wrdecoder.png
Attachment 11: 1.png
1.png
Attachment 12: 2.png
2.png
Attachment 13: 3.png
3.png
Attachment 14: 4.png
4.png
Attachment 15: 5.png
5.png
Attachment 16: 6.png
6.png
Attachment 17: 7.png
7.png
Attachment 18: 8.png
8.png
Attachment 19: 9.png
9.png
Attachment 20: 10.png
10.png
Attachment 21: 11.png
11.png
Attachment 22: 12.png
12.png
Attachment 23: 13.png
13.png
Attachment 24: 18hOdd.png
18hOdd.png
Attachment 25: even18h.png
even18h.png
Attachment 26: cja_bias.png
cja_bias.png
Attachment 27: cja_stats.png
cja_stats.png
Attachment 28: cja_temp.png
cja_temp.png
Attachment 29: pulser_peaks1.png
pulser_peaks1.png
Attachment 30: pulser_peaks2.png
pulser_peaks2.png
  102   Wed Nov 27 16:33:50 2019 NHNew Bias & Waveforms

HV core (-160 V) now attached to bottom 3 FEEs (with grounded kapton)

DSSDs bias OK, leakages unchanged

No appreciable change in waveforms

Some estimates

aida01/aida05/aida09 VHF noise: 5 channels frequency = 10 MHz ?

All FEEs:
Oscillation at ~25-35 channels = 1.4 to 2 MHz ?
Varies a bit with each FEE.

No obvious issue with upper FEEs unsure why 10 MHz is present - fee09 has been replaced but showed it before as well.

Lower frequency noise on every FEE may be ground noise from e.g. switching PSU?

No more access to S4 to check thoroughly again.

 

 

 

 

Attachment 1: waves_new_odd.png
waves_new_odd.png
Attachment 2: waves_new_even.png
waves_new_even.png
  101   Mon Nov 25 13:25:51 2019 NHWaveforms
> AIDA Waveforms immediately after beam at 21.11.19 8am or so.

Update 25.11.19 with all FEE64 waveforms added

Things to note:
aida01, aida05, aida09 are all on TOP - show VHF noise, esp. aida01?
TOP is where HV is connected (-ve core)

Everything noisy in general though
Attachment 1: wave_odd.png
wave_odd.png
Attachment 2: wave_even1.png
wave_even1.png
  100   Fri Nov 22 08:55:31 2019 NHWaveforms
AIDA Waveforms immediately after beam at 21.11.19 8am or so.
Attachment 1: waves_21119_even.png
waves_21119_even.png
Attachment 2: 21119_waves_odd.png
21119_waves_odd.png
Attachment 3: waves_21119_even_z.png
waves_21119_even_z.png
Attachment 4: waves_21119_odd_z.png
waves_21119_odd_z.png
  99   Wed Nov 20 20:36:14 2019 CA, CB, DK, NH20/21 November Night Shift
21:35 (NH) - A few merger timestamp errors were appearing - Decided to powercycle all FEE64s, all checks OK.
 No Screenshot due to forgetting before I closed merger 
 FEEs were often not stopping properly and needing merger reset as well.

Beam Plan:

40Ar 300 MeV/u Primary Beam
and/or
34Si Fragment from Ar

22:45

FEES were restart near these timestamps in the TapeData, evidently
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:17 R1_361
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:36 R1_362

Run number was kept the same (Tape Server was not stopped @ FEE restart?)

All system wide checks are okay (except of course the master clock fails)
Temps look okay (attachment 1, dated 21/Nov by mistake)
Leak currents all around 0.9 uA as before (attachment 2)
Stats are behaved. We don't have beam yet (attachment 3)
Merger and tape server happy (see attachment 4)
Data rate histos while beam is still off (see attachment 5)

==Nov 21==

0:00
TD noted that the data rates have gone up at some point (about double).  
Now we increased the slow comparator thresholds from 0xa to 0xb (data rate drops from 24 Mb/s to 16 Mb/s).
We increased it again from 0xb to 0xc, but near that time the beam was also turned on, and the data rate was about 25 Mb/s.

Beam was turned on near R1_472 or a bit earlier.

TD requested waveforms, but at the moment they are not appearing.  Will enable WFs after a condition change.

Beam seen in all three DSSSDs, see attachment 6.  Beam was turned off before I could get more screencaps.

Degrader change ... near R1_481

00:32 waveforms now enabled under Run Control.  They also appear to be enabled on ASIC1 undre LED and Waveform controls.  However, we are unable to see the waveforms in the relevant histograms.

00:44 3 g/cm^2 inserted R1_488

00:45 3.2 g/cm^2 degrader in R1_489.

1:00 Several degraders and combinations are being tested since last note.

1:10 It seems 1 g/cm^2 of energy loss is not accounted for by LISE++.  Trying to understand the reason.

1:28 Satisfied with a total degrader thickness of 2.6 g/cm^2 -- seems to calculate that fragments should implant mainly in DSSSD2?  Issue seems to be resolved updating the LISE++ file.
  Near R1_507

Can see the implant samples attached as #7 and #8

1:43 According to the stats, most of the beam energy is being deposited in DSSSD1.  See #9.
  Present beam burst repetition is around 0.1 Hz (each ~10 sec)

2:00 better example of stats histos (see #10)

2:45 tuning the FRS degraders (ones for RIB separation rather than AIDA implantation depth).

3:17 Just started fragments.  Disregard previous filenames with "34Si".  Finally no degrader in the FRS was needed to get Z separation.
  R1_559
  As the DAQs are not synchronized, a quick call to the BNC PB-5 pulser shifiting from 2 -> 50 -> 2 Hz was used to make a signal burst.
  S4 degrader is 5.2 g/cm^2

3:22 Updated biases in Google Sheets.

Having trouble seeing the fragments in the AIDA histograms.  Since we just started fragments, zero the histos.

3:35 Attachment 11, with 34Si fragments as Stats.  Difficult to click Update on the Rates at the right time.  
   From the beam monitors, fragments should be at around 3000 cps.

3:45 (Up to now, some optics tuning with slits, etc)

3:46 Optics settings accepted, AIDA at R1_572, accept this condition for data runs on ~34Si fragments.

3:47 Sent a pulser burst 2 -> 50 -> 2 Hz

3:50 zero histograms again.
   Temperatures attachment 12
   Biases attachment 13
   Statistics attachment 14
   Merger / Tape Server attachment 15

3:56   
   StatHistos attachment 16

4:09 FRS starts new run #56, AIDA R1_583.  Pulser 2-> 50 -> 2 Hz

4:56 Stat histograms in attachment 17.  Not zeroed since noted at 3:50
Attachment 1: 21-11-temps.png
21-11-temps.png
Attachment 2: 20-11-biases.png
20-11-biases.png
Attachment 3: 20-11-stats.png
20-11-stats.png
Attachment 4: 20-11-merger-ts.png
20-11-merger-ts.png
Attachment 5: 20-11-statshistos.png
20-11-statshistos.png
Attachment 6: 21-11-layout5histos.png
21-11-layout5histos.png
Attachment 7: 34si-implants-130-1.png
34si-implants-130-1.png
Attachment 8: 34si-implants-130-2.png
34si-implants-130-2.png
Attachment 9: 34si-stats-histos.png
34si-stats-histos.png
Attachment 10: stats-histos-2am.png
stats-histos-2am.png
Attachment 11: 34si-fragments-stats.png
34si-fragments-stats.png
Attachment 12: temperatures-fragments.png
temperatures-fragments.png
Attachment 13: biases-fragments.png
biases-fragments.png
Attachment 14: stats-fragments.png
stats-fragments.png
Attachment 15: merger-tapeserver-fragments.png
merger-tapeserver-fragments.png
Attachment 16: stathistos-fragments.png
stathistos-fragments.png
Attachment 17: stathistos-5am.png
stathistos-5am.png
  98   Tue Nov 19 21:22:06 2019 CA, CB, DK, NH19/20 November 2019 - overnight shift
19 November

Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.

Tomorrow fragments could be expected.

DSSD / FEE64 pairing as follows

DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream

New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023

DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.


System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.

MBS transfer OK. See attached.

Good event statistics OK. See attached.

FEE64 temperatures OK. See attached.

Merger and data rate OK. See attached.

Waiting for beam.



20 November

01:07 - Still no beam. 

System wide checks OK. 
Stats OK. 
Temps OK. 
MBS OK. 
Merger and data rate OK.


02:35  - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.

No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.

DAQ & Merger going OK. Writing to Nov19/R1

No beam yet.

Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.

~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.

03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.

04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.

~04:30 Moved to 6.2 g/cm2 degrader. See attached.

~04:45 Moved to 5.8 g/cm2. See attached.

~05:09 Moved to 6.4 g/cm2. See attached.

05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14

05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.

06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today.
Attachment 1: Bias2235.png
Bias2235.png
Attachment 2: Rates2236.png
Rates2236.png
Attachment 3: 19112241MBS.png
19112241MBS.png
Attachment 4: 19112242Stats.png
19112242Stats.png
Attachment 5: 19112244Temperatures.png
19112244Temperatures.png
Attachment 6: 19112246Merger.png
19112246Merger.png
Attachment 7: 20110239.png
20110239.png
Attachment 8: 20110240DataTransfer.png
20110240DataTransfer.png
Attachment 9: 6gdegrader.png
6gdegrader.png
Attachment 10: 6.2degrader.png
6.2degrader.png
Attachment 11: 5.8degrader.png
5.8degrader.png
Attachment 12: 6.4degrader.png
6.4degrader.png
Attachment 13: 18h_0530.png
18h_0530.png
Attachment 14: 18H_0528.png
18H_0528.png
Attachment 15: 1gcm2degrader.png
1gcm2degrader.png
  97   Wed Nov 13 15:38:20 2019 NH[HowTo] MBS for AIDA (Nov 19)
For MBS integration AIDA forwards its data to an MBS Foreign Data Receiver (FDR), currently the PC assigned for this task is x86l-94.

To start the MBS system do the following:

Connect to the MBS FDR: ssh despec@x86l-94 (Ask local for password)

cd to the AIDA directory: cd mbsrun/nov19/aida_to_mbs

Start mbs: mbs

Startup the receiver: @startup

Now open the MBS relay for MIDAS (far right icon on the top)

Check both terminals (Relay & MBS) report a connection

In another terminal ssh into the FDR and run rate to monitor the data rate in MBS

For data to be transferred the Tape Server must be GOING but can be in No Storage mode if no MIDAS storage is required.

--

For experiments MBS file saving is handled by the DESPEC time sorter which merges all the subsystems together.
For testing you can write files from x86l-94 using the following commands

connect rfio XXX -disk
Where XXX is an Linux PC (ask local for a PC)

open file /path/to/file_ first=1 size=2000 -auto -rfio
This opens the file, writing in 2GB chunks

clo file
This will close the file when you are done

--

It is possible to restart the MBS relay at any point if the connection seems to have failed, also confirm the Tape Server is GOing
  96   Wed Nov 13 09:10:37 2019 CA, NH, OH13th November 2019
10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   
Attachment 1: 191113_pulserpeaksOdd.png
191113_pulserpeaksOdd.png
Attachment 2: 191113_pulserpeaksEven.png
191113_pulserpeaksEven.png
Attachment 3: 191113_goodevents.png
191113_goodevents.png
Attachment 4: 191113_bias.png
191113_bias.png
  95   Wed Nov 13 01:54:00 2019 TDAida10 Unusual Pulser Peak

Double peaking normally indicates periodic baseline variations with period comparable to shaping time

Quote:

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

 

  94   Tue Nov 12 14:10:18 2019 NH, OH, CAAida10 Unusual Pulser Peak

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

Attachment 1: aida10-pulser-121119.png
aida10-pulser-121119.png
Attachment 2: pulsers121119.png
pulsers121119.png
Attachment 3: pulsersodd121119.png
pulsersodd121119.png
  93   Tue Nov 12 13:59:07 2019 NH, OH, CAAwfully noisy waveforms
Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8
Attachment 1: awfulwaves121119.png
awfulwaves121119.png
Attachment 2: 191112_GoodEvents.png
191112_GoodEvents.png
Attachment 3: 191112_StatsPostLoad.png
191112_StatsPostLoad.png
Attachment 4: pulsersettings191112.png
pulsersettings191112.png
Attachment 5: pulserpeaksOdd.png
pulserpeaksOdd.png
Attachment 6: pulserpeaks_Even.png
pulserpeaks_Even.png
Attachment 7: merger191112.png
merger191112.png
Attachment 8: tapeservice191112.png
tapeservice191112.png
  92   Tue Nov 12 13:22:19 2019 OH, NH, CAAIDA09 Replacement
ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2
Attachment 1: FEE-SerialNumber.txt
FEE#	Messanie Serial No.	MAC Address
4	026-1690101-0016	d8-80-39-41-f6-b7
8	026-1690101-0013	d8-80-39-41-ba-2b
12	026-1690101-0030	d8-80-39-41-cf-ac
2	026-1690101-0011	d8-80-39-41-ba-22
6	026-1690101-0035	d8-80-39-41-ee-72
10	026-1690101-0032	d8-80-39-41-ee-10
1	026-1690101-0027	d8-80-39-41-ba-8a
5	026-1690101-0008	d8-80-39-41-d7-cc
9	026-1690101-0015	d8-80-39-41-f6-ee
7	Not checked	d8-80-39-41-b4-0c
11	026-1690101-0006	d8-80-39-41-f6-5a
3	026-1690101-0019	d8-80-39-41-d8-21
		
		
		
Dead FEE	Messanie Serial No.	MAC Address
ASIC 1 Dead	026-1690101-0005	d8-80-39-42-0d-0c
  91   Mon Nov 11 10:34:19 2019 NHAlpha Analysis (11.11.19)

Analysis of approx 20GB of data since Thursday night (not continuous - estimate of hours incominG)

Figures 1-3: Energy Front vs Energy Back (No F-B energy gate)

Figures 4-6: 2D hit pattern

Figures 7-9: 1D hit pattern

aida10 does not seem to be reading out still, unsure of reason? Looks fine with pulser.
Slow comparator not setting or unusual gain?

Otherwise good

Attachment 1: DSSD1-Energy.png
DSSD1-Energy.png
Attachment 2: DSSD2-Energy.png
DSSD2-Energy.png
Attachment 3: DSSD3-Energy.png
DSSD3-Energy.png
Attachment 4: DSSD1-2DHP.png
DSSD1-2DHP.png
Attachment 5: DSSD2-2DHP.png
DSSD2-2DHP.png
Attachment 6: DSSD3-2DHP.png
DSSD3-2DHP.png
Attachment 7: DSSD1-1DHP.png
DSSD1-1DHP.png
Attachment 8: DSSD2-1DHP.png
DSSD2-1DHP.png
Attachment 9: DSSD3-1DHP.png
DSSD3-1DHP.png
  90   Thu Nov 7 10:22:26 2019 Nic, PatrickReply: WR Timestamps
> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > > 
> > > HDMI cables in aida09 checked and good.
> > 
> > The problem would be the cable , one end or the other. 
> > I think ( if I recall ) a setup would restart the WR decoder.
> > 
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
> 
> Cable will be replaced once a spare is available.
> 
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
> 
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:

aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000

aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
  89   Tue Nov 5 11:15:41 2019 NH PJCSReport: aida10 database corruption

If ypou now check the actual values in the aida10 Options screen are the correct ?

It can happen that corruptions once loaded are propagated when Options are generally saved. 

Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?

Quote:

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

 

  88   Tue Nov 5 10:32:21 2019 NH PJCSReport: aida10 database corruption

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

  87   Tue Nov 5 10:02:45 2019 NH PJCSReport: aida10 database corruption

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

  86   Mon Nov 4 13:17:12 2019 NHReport: aida10 database corruption

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

Attachment 1: db_working.txt
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
Attachment 2: db_corrupt.txt
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&0x0006dead&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&Aida_Hist_H_Enable&&DataAcqPgm&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 0
Aida.shift string 1
TS_SYNC_PHASE string 1
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 14
0x0006dead string 0x0000
Stat.channels string 150
Aida_GroupBase string 1
Rate.channels string 150
RunNumber string 2
Aida_Hist_D_Enable string 1
ASIC.settings string undefined
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
Stat.shift string 6
Aida.Wchannels string 1020
Include.Aida string 1
Aida.channels string 32768
Aida_Hist_V_Enable string 0
Aida_Hist_H_Enable string 0
DataFormat string AIDA
DataAcqPgm string AidaExecV9
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
  85   Mon Nov 4 08:48:18 2019 NHAlpha Status 04.11.2019

Alpha run has been running over the weekend. Stats looking positive with notable exception that aida10 is not sending much data. Will attempt a powercycle.

Temperatures and Leakage Currents very good to excellent

Noted that the slow rate of aida10 seems to be slowing down the rate data is sent to tape - perhaps being buffered until an ADC event arrives?

Update 05.11.2019

Most rates OK, aida10 only recorded 1 event over night, aida09 spectra had a number of weird channels.
Performed ASIC check/load, aida10 found another 201 events but mostly in the HEC channel (as did all FEEs)
All other FEEs look very nice however

 

15:44 CET: Check/load performed as aida07 rate went very high (in HEC it seems) - now back to 0

 

Update 07.11.2019
Run stopped to move DESPEC platform, overnight the USB relay disconnected/reconnected powering down the FEEs (probably someone knocked the interlock wire)

 

Attachment 1: alphas-bias.png
alphas-bias.png
Attachment 2: alphas-temp.png
alphas-temp.png
Attachment 3: alpha-aida10.png
alpha-aida10.png
Attachment 4: stats-0511.png
stats-0511.png
Attachment 5: spec-0511.png
spec-0511.png
  84   Fri Nov 1 18:09:03 2019 CA, TD, NHSummary - 29.10-1.11.19
> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom
  83   Fri Nov 1 15:45:04 2019 CA, TD, NHSummary - 29.10-1.11.19
>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.
  82   Fri Nov 1 15:24:40 2019 CA, TD, NHSummary - 29.10-1.11.19
3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc
  81   Fri Nov 1 14:17:15 2019 Nic, PatrickReply: WR Timestamps
> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > 
> > HDMI cables in aida09 checked and good.
> 
> The problem would be the cable , one end or the other. 
> I think ( if I recall ) a setup would restart the WR decoder.
> 
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Cable will be replaced once a spare is available.

Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.

WR rate was chosen by Vic I believe, I am unsure of reasoning myself.
  80   Fri Nov 1 10:46:13 2019 CA, TD, NHFriday 1st November 2019
11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding
Attachment 1: biaswfoil.png
biaswfoil.png
Attachment 2: statswfoil.png
statswfoil.png
Attachment 3: tempwfoil.png
tempwfoil.png
Attachment 4: biaswofoil.png
biaswofoil.png
Attachment 5: statswofoil.png
statswofoil.png
Attachment 6: tempswofoil.png
tempswofoil.png
  79   Fri Nov 1 09:30:17 2019 NHDatabase error stopping run?
Strange error about Database options appearing during run stop.
Does not seem to affect stopping/starting/data?
Attachment 1: ErrorStopDatabase.png
ErrorStopDatabase.png
  78   Thu Oct 31 19:02:42 2019 PatrickWR Timestamps
> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
  77   Thu Oct 31 17:13:54 2019 NHAlpha Run
Beginning Alpha run

Thresholds set to 100 (LEC/Slow) 255 (LEC/Fast) 2 (HEC)

Pulser is OFF

Recording to MIDAS: /TapeData/31Oct19/R1

Recording to MBS: /lustre/gamma/nhubbard/AIDA/Alphas311019_

Disk rate approximately 0 as expected.

01.11.19 19:11 CET: FEE10 Stat histogram was empty - performed a check/load and now rare events come in - Merger also emitted 10 blocks to storage quickly (presumably waiting on FEE10 data)
All FEEs now slowly showing alpha data
  76   Thu Oct 31 16:43:03 2019 NH CA TDMerger & MBS Performance
Testing merger with 3 DSSDs connected, with thresholds of 10 (slow comparator), 2 (HEC) and 255 (fast discriminator)

Merger handling rate of 1.6 million events per second comfortably. Forwarding to MBS fine.

Network usage: 130 Mbps (AIDA network) and 20 Mbps (MBS network)
Attachment 1: MergerStats.png
MergerStats.png
Attachment 2: MBSRates.png
MBSRates.png
Attachment 3: Mergernetworking.png
Mergernetworking.png
Attachment 4: MergerStatistics.png
MergerStatistics.png
  75   Thu Oct 31 15:24:35 2019 TDwaveform spectra issues?
Attachment 1: 20.png
20.png
Attachment 2: 21.png
21.png
Attachment 3: 22.png
22.png
Attachment 4: 23.png
23.png
Attachment 5: 30.png
30.png
Attachment 6: 31.png
31.png
Attachment 7: 32.png
32.png
Attachment 8: 33.png
33.png
Attachment 9: 40.png
40.png
Attachment 10: 41.png
41.png
Attachment 11: 42.png
42.png
Attachment 12: 43.png
43.png
  74   Thu Oct 31 14:27:06 2019 NHWR Timestamps
All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.
Attachment 1: WRTimes.png
WRTimes.png
  73   Thu Oct 31 14:23:37 2019 TDaida09 asic 1 no data readout?
Attachment 1: 10.png
10.png
Attachment 2: 11.png
11.png
Attachment 3: 12.png
12.png
Attachment 4: 13.png
13.png
  72   Thu Oct 31 10:33:34 2019 TD[How To] Start TclHttpd server on Raspberry Pi
1) Login to Raspberry Pi 

ssh pi@nnrpi2

2) Kill any instances of the TclHttpd server

ps -ef | grep TclHttpd
kill -9 <pid>

3) Start server

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &
  71   Thu Oct 31 09:36:37 2019 TDRIKEN LayOut directory
Attachment 1: LayOut.tar.gz
  70   Thu Oct 31 09:00:58 2019 CA, TD, NHThursday 31 October
10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload
Attachment 1: 1.png
1.png
Attachment 2: 31OctTemp1119.png
31OctTemp1119.png
Attachment 3: stats31Oct1128.png
stats31Oct1128.png
Attachment 4: 50stats.png
50stats.png
Attachment 5: 20stats.png
20stats.png
Attachment 6: 10stats.png
10stats.png
Attachment 7: 1335rates.png
1335rates.png
Attachment 8: 1337_stats.png
1337_stats.png
Attachment 9: 1341_pulser.png
1341_pulser.png
Attachment 10: 1345pulser.png
1345pulser.png
Attachment 11: 1348_18W.png
1348_18W.png
Attachment 12: 1350_18W.png
1350_18W.png
Attachment 13: 1728_stats.png
1728_stats.png
Attachment 14: 1730_stats.png
1730_stats.png
  69   Wed Oct 30 11:51:08 2019 CA, TD, NHDetector biasing
12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.
       
Attachment 1: bias30Oct.png
bias30Oct.png
Attachment 2: 30Octbias2.png
30Octbias2.png
Attachment 3: hvsketch.pdf
hvsketch.pdf
  68   Wed Oct 30 08:06:49 2019 CA, TD, NHDSSSD stack
Ribbon cables fitted with kapton PCB couplers -> 3 isolated + 1 non-isolated for each cable length/DSSSD

Ribbon cable lengths:

without kapton PCB coupler (cm) - 40, 47, 51.5

with kapton PCB coupler (cm) - 50.5, 57.5, 62

gap between plastic and dsssd - 15 mm

length between top of snout and dsssd - 5cm
Attachment 1: IMG_20191030_091729.jpg
IMG_20191030_091729.jpg
Attachment 2: IMG_20191030_091734.jpg
IMG_20191030_091734.jpg
Attachment 3: IMG_20191030_091925.jpg
IMG_20191030_091925.jpg
Attachment 4: IMG_20191030_091930.jpg
IMG_20191030_091930.jpg
Attachment 5: IMG_20191030_102326.jpg
IMG_20191030_102326.jpg
Attachment 6: IMG_20191030_102329.jpg
IMG_20191030_102329.jpg
  67   Mon Oct 28 17:02:11 2019 TDAIDA @ DESPEC cable length estimates
AIDA snout length 37cm (from end of AIDA snout support assembly)

downstream scintillator (assembly depth 1cm)   36cm

downstream dsssd (#1)                          35.2cm

                 (#2)                          34.3cm

                 (#3)                          33.4cm

upstream dsssd   (#4)                          32.5cm

upstream scintillator (assembly depth 1cm)     31.7cm

Cable lengths required are therefore

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 22.5 = 54.35cm 21.4"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 16.5 = 47.45cm 18.7"

                 (#3)           33.4 - 10.5 + 2.15 + 5 + 10.5 = 40.55cm 16.0"

upstream dsssd   (#4)           32.5 - 10.5 + 2.15 + 5 + 4.5 = 33.65cm 13.2"

length of RH Kapton PCB coupler = 10.5cm (Carlo can you check please?)

turning the corner at the base of the AIDA snout support assembly 2.15+5=7.15cm
(see Peter's drawing)

distances to FEE64s 4.5, 10.5, 16.5 and 22.5cm
(see Peter's drawing)

f we use 3x DSSSDs then the cable lengths required will be as follows:

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 16.5 = 48.35cm 19.0"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 10.5 = 41.45cm 16.3"

upstream dsssd   (#3)           33.4 - 10.5 + 2.15 + 5 + 4.5 = 34.55cm 13.6"
  66   Mon Sep 16 12:24:46 2019 NHAIDA Monitoring

https://docs.google.com/spreadsheets/d/1fZBybHodSvacUdKpDM_VKznqTrh49CKnqKVQVMEDWGs/edit?usp=sharing

A spreadsheet of ambient conditions and water temperature in S4 for AIDA.
Will show the distance to danger points (interlock shutdown and condensation on the pipes)

  65   Fri Aug 16 15:09:05 2019 NHAida Waves (All FEEs)

Included waves for all 12 FEEs

FEES 1, 5, 9 all show this HF noise on it, and are all located at the top of the AIDA mount.
Connections: HV from DSSD (-ve) is in FEE9
Pulser input is in FEE9 (non-inverted)

Resistance between FEE cooling plate and LEMO connector for all 3 FEEs is a few Ohms, as is resistance between copper braid on DSSD cable and the LEMO connector.

All FEEs show some noise but mainly those 3 and all four with DSSD. Will try to check DSSD connections in case something is messed up.

FEE layout for reference: (Beam's perspective)

                              FEE9
                              FEE5
                              FEE1

FEE10 FEE6 FEE2                FEE4 FEE8 FEE12

                              FEE3
                              FEE7
                              FEE11

Attachment 1: Pulse-AIDA01.png
Pulse-AIDA01.png
Attachment 2: Pulse-AIDA02.png
Pulse-AIDA02.png
Attachment 3: Pulse-AIDA03.png
Pulse-AIDA03.png
Attachment 4: Pulse-AIDA04.png
Pulse-AIDA04.png
Attachment 5: Pulse-AIDA05.png
Pulse-AIDA05.png
Attachment 6: Pulse-AIDA06.png
Pulse-AIDA06.png
Attachment 7: Pulse-AIDA07.png
Pulse-AIDA07.png
Attachment 8: Pulse-AIDA08.png
Pulse-AIDA08.png
Attachment 9: Pulse-AIDA09.png
Pulse-AIDA09.png
Attachment 10: Pulse-AIDA10.png
Pulse-AIDA10.png
Attachment 11: Pulse-AIDA11.png
Pulse-AIDA11.png
Attachment 12: Pulse-AIDA12.png
Pulse-AIDA12.png
  64   Thu Aug 15 14:09:39 2019 NHAIDA Waveforms

Noise seemed less strong today, unsure of cause. I had re-taped snout as tape was not sticking well.

Figures 1-4: Waves for all 4 FEEs connected to the DSSD
Figures 5-8: Zoomed in on region 400-600
Figures 9-10: Rates in DSSD

Note in particular fee09 has a clear noise pickup somewhere. Also appears when no DSSD voltage.
Peak distance = 6 channels = 16.6 MHz.

Fee09 is connected to HV bias core (-160 V)
However unplugging the HV cable from it shows no difference.

Must be related to something else.

Update 16:29 CEST:
Distance is more like 5 channels (20 MHz)
Fee09 picks it by far most strongly but all fees you can make it out
Additionally can see a slower waveform especially on other fees... approximately 40 channels = 2.5 MHz

Checked with multimeter that low resistance path between FEE cooling plate and ground LEMO connector
also low resistance between ribbon cable copper shield and ground LEMO connector.

Noise seems to appear even if pulser cable unplugged at pulser end or when attentuation is increased weaking pulse.

 

Attachment 1: aida09-wave-all.PNG
aida09-wave-all.PNG
Attachment 2: aida10-wave-all.png
aida10-wave-all.png
Attachment 3: aida11-wave-all.png
aida11-wave-all.png
Attachment 4: aida12-wave-all.png
aida12-wave-all.png
Attachment 5: aida09-wave-zoom.png
aida09-wave-zoom.png
Attachment 6: aida10-wave-zoom.png
aida10-wave-zoom.png
Attachment 7: aida11-wave-zoom.png
aida11-wave-zoom.png
Attachment 8: aida12-wave-zoom.png
aida12-wave-zoom.png
Attachment 9: RateHists.png
RateHists.png
Attachment 10: Rates.png
Rates.png
  63   Wed Aug 14 14:40:18 2019 NHHigh Frequency Noise in AIDA

Figures of intense high frequency noise pickup in AIDA.

Will investigate source in S4.

Edit: Period is approx 40 cycles / approx 2.5 MHz (?)

Attachment 1: HF.png
HF.png
Attachment 2: HFZ.png
HFZ.png
  61   Tue Aug 13 13:55:45 2019 NHNew Merger/MBS Test Runs

AIDA Finally working again so can test VP's added WAVE histogramming mode.

Fig 1. Waveforms for FEE12 ASIC 1. Sample rate = 10, Threshold = 10500
n.b. lots of "blank" wave channels for unknown reason, all low-energy channels show pulser so it's not a DSSD signal problem?

Fig 2. Pulser spectrum for FEE12 ASIC 1. (10 peaks 0.1 -> 1 V)

Fig 3. Waveforms for FEE10 ASIC 3.

n.b. WAVE channels frequently stopped updating, needed start/stop DAQ or restart WAVE readout option. Unsure if it's related to high rate/odd signals.
Might be related to issue in FEE12 too. WAVE good events statistics lower than empty FEEs.
All rates quite high, looks quite noisy at the moment.

Fig 4. Pulser of FEE10 ASIC 3.

Note that FEE10.3 has poor peaks, in WAVE we see:
Some normal looking pulses which are merely noiser than other channels.
Some pulses which look very odd (baseline really low, etc).

Will recheck ground for noise issues, but the huge peaks are confusing to me. (3.5.W, etc)

Quote:

New merger has been worked on by VP which fixes the timewarp issues and MBS integration.

Currently no WAVE capture is supported, VP will re-add WAVE histogramming for startup testing.

10.05.2019 - Pulser walkthrough
-> Pulser rate 50 Hz
-> 1 minute per 0.1 V (0.1 to 1 V)
-> extra time on 1 V

With this 95% of channels calibrate easily (good statistics)
Issue: ~16 channels (most of ASIC3) in FEE10 are horrificly noisy. Not calibrated

Fig 1. Pulser of FEE1 (no DSSD)
Fig 2. Pulser of FEE9 (DSSD)
Fig 3&4. Pulser of FEE10 noisy channels

Future TODO: Will investigate WAVE data and look if issue can be found.

20.05.2019 - Alpha walkthrough
-> Pulser off
-> FEE10ASIC3 has readout off & discriminator masked
-> Slow threshold 0x64
-> Fast threshold 0xff
-> HEC threshold 0x2

File is nyx/gamma/nhubbard/AIDA/alpha_bg_200519_

Rate is approx. 1MB/s still of WR timestamp data.

(NB, MIDAS Alpha walkthrough before beamtime I see no alpha events in!)

Fig 5: Temps
Fig 6: Stats
Fig 7: Merger Stats
Fig 8: MBS Stats

 

Attachment 1: AIDA_FEE122-ASIC1.png
AIDA_FEE122-ASIC1.png
Attachment 2: Pulser-FEE12-ASIC1.png
Pulser-FEE12-ASIC1.png
Attachment 3: AIDA-FEE10-ASIC3.png
AIDA-FEE10-ASIC3.png
Attachment 4: Pulser-FEE10-ASIC3.png
Pulser-FEE10-ASIC3.png
Attachment 5: Rates-WAVE.png
Rates-WAVE.png
Attachment 6: Rates-good.png
Rates-good.png
  60   Tue Aug 13 09:17:02 2019 NHAIDA 12/08/2019

The other 6 FEEs were powered on one at a time to check no issues. None found.
All 12 FEEs plugged back into the PSU.

Note: The FEE temperatures reported on Friday were before a Reset/Setup of the DAQ, which means the ASICs run very fast and the hence run hotter.
Reset/Setup shows more sustainable temps (attached)

These temps are similar to those observed in Sep 2018 but warmer by ~4 C than those observed Dec-Jan and April.

Attachment 1: AIDATemps-1208191457.png
AIDATemps-1208191457.png
  59   Fri Aug 9 12:33:14 2019 NHAida Test 9/8/19 - FEES 1-6

The water issue has been resolved - merely incorrect temperature guages (replaced). Water is 20 C as required.

S4 conditions at 13:12 CEST

26.5 C / 47.2 % / Td = 14.3 C

Based on email by TD doing the following procedure to  ensure FEEs are OK after water issue

1) disconnect the power cables of *all* FEE64s at the FEE64 PSUs
2) re-connect power cable, power-up and check/test the FEE64s
  *one at a time* until stable operatinmg temperature is achieved
... say 30mins?

13:34 CEST: All be FEE1 unplugged. Powering on!

Vertex temp read slightly warm (67 C) so powered off. Waiting for a while for water to circulate since off for a while...

15:01 CEST: FEE1 Powered on again, Vertex reading 67 C again but it's stable and slowly going down. Watching and seeing but I think it's OK.
15:35 CEST: FEE1 has been stable around 67 C all the time. Other temps fine. Presuming this isn't a major concern. (Perhaps related to reasonably warm/humid conditions?)

15:47 CEST: FEE2 Powered on.
16:02 CEST: Temperature stable, not exceeded 60 C.

16:07 CEST: FEE3 Powered On.
16:17 CEST: Temperature stable, peaked at 58.44 C

16:21 CEST: FEE4 Powered On.
16:40 CEST: Both FPGA and ASIC reading warm: 65.5 C and 58 C. Seeming to not be rising anymore.

16:41 CEST: FEE5 Powered On.
16:53 CEST: Temperature stable, at 64 C

16:56 CEST: FEE6 Powered On.
17:12 CEST: FPGA temp is ok but ASIC warm. Tmax ASIC = 56.88, FPGA = 55.38, PSU = 27.44

17:15 Turning FEEs and Water off for the weekend
Will test 7-12 on Monday.
Might be cooler in S4.

Cooling seems to be working but temps hotter than before.

 

  58   Mon Aug 5 12:57:48 2019 NHS4 Conditions

Temp: 26 C
Humidity: 50%
Dew Point: ~15 C

There are two temperature dials outside S4, one shows exactly 20 C and one shows nearer 16 C.

Trying to clarify this discrepancy as 16 C is too cold at the moment.

In contact with developer of cooling system to confirm apparent 16 C after the heat exchanger

Today's conditions:

Temp: 26.9 C
Humidity: 48.7%
Dewpoint 15.2 C

  57   Wed Jul 10 10:46:05 2019 NHAIDA/Water 10.07.2019

S4 Conditions:

25.2 C, 35.3% RH, Td = 8.8 C

Water temperature: 20 C

-

Rotating the dewpoint sensor on the pipe seems to have made it switch over to solid red = humidity OK.
Waiting confirmation to see if we should check a FEE (as per PJCS suggestion) before water is turned back on.
Will check the outside copper after lunch.

13:04 : Copper seems warm and dry, humidity sensor still solid red.

  56   Mon Jul 8 16:02:52 2019 NH, PJCSAida 28/6/2019 - WATER OFF

It occurs to to wonder if there is now water inside the modules. If the metal of the cooling plates is damp on the outside then it is safe to assume there is water on the inside too . Can I suggest that one  of the modules is dismounted, disassembled and checked?

Patrick

Quote:

The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.

Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.

 

  55   Mon Jul 1 09:42:16 2019 NHAIDA - 01/07/2019

Water is still off.

Checked dewpoint sensor still flashing red light (indicating an error). Pipes warmed. Not what I expected. Possible fault?

I checked S4's ambient temp/humidity with a home-brought sensor: Reports 27 C/50% RH (Dew Point = 16 C)
Water temperature is set to exactly 20 C (there's a second dial showing ~18 C but I am unsure if that's the same water supply, I will ask)

Checking the dewpoint PSU with a multimeter shows a voltage of 16 V.
The PSU itself is rated for 12 V output, and the humidity sensor expects 24 V.
Regardless of which voltage 16 V seems incorrect. Might be causing issues.

Humidity is a bit lower now (4pm) the sensor has been in all day, 28 C/35% RH (Dew Point = 11 C) which isn't an issue.

Water is still off until I am sure what to do, or until the dewpoint sensor stops complaining.

 

  54   Fri Jun 28 15:05:49 2019 NHAida 28/6/2019 - WATER OFF

The humidity sensor hasn't reported humdities under 90% yet. In order to try and help dry the pipes and turn AIDA the water has been turned off the weekend.
Sunday temperatures might reach 40 C which means it may struggle anyway.

Will consider trying to get a temperature/humidity monitor for S4 to check if the environment is fine too.

  53   Tue Jun 4 09:19:10 2019 NHAIDA Interlock

Currently AIDA's interlock is off (no lights) and hence AIDA cannot be powered on. Under investigation - power supply has been replugged but seems ok.

Update: It seems the interlock is correctly stopping AIDA from being powered on due to high humidity on the cooling lines. Am currently trying to verify the temperature with GSI technical staff

  52   Mon May 20 13:51:45 2019 NHHowTo Verify WR Times

The latest version of MIDAS has a new page to check the WR timestamp of each FEE.

In AIDA Hardware control click: GSI White Rabbit Control
In expert options click: Collect all Timestamps
Verify every FEE has a good timestamp.

Attachment 1: HowTo-WR1.png
HowTo-WR1.png
Attachment 2: HowTo-WR2.png
HowTo-WR2.png
Attachment 3: HowTo-WR3.png
HowTo-WR3.png
  51   Mon May 20 13:33:06 2019 NHNew Merger/MBS Test Runs

New merger has been worked on by VP which fixes the timewarp issues and MBS integration.

Currently no WAVE capture is supported, VP will re-add WAVE histogramming for startup testing.

10.05.2019 - Pulser walkthrough
-> Pulser rate 50 Hz
-> 1 minute per 0.1 V (0.1 to 1 V)
-> extra time on 1 V

With this 95% of channels calibrate easily (good statistics)
Issue: ~16 channels (most of ASIC3) in FEE10 are horrificly noisy. Not calibrated

Fig 1. Pulser of FEE1 (no DSSD)
Fig 2. Pulser of FEE9 (DSSD)
Fig 3&4. Pulser of FEE10 noisy channels

Future TODO: Will investigate WAVE data and look if issue can be found.

20.05.2019 - Alpha walkthrough
-> Pulser off
-> FEE10ASIC3 has readout off & discriminator masked
-> Slow threshold 0x64
-> Fast threshold 0xff
-> HEC threshold 0x2

File is nyx/gamma/nhubbard/AIDA/alpha_bg_200519_

Rate is approx. 1MB/s still of WR timestamp data.

(NB, MIDAS Alpha walkthrough before beamtime I see no alpha events in!)

Fig 5: Temps
Fig 6: Stats
Fig 7: Merger Stats
Fig 8: MBS Stats

Attachment 1: 20190522-PulserFEE1.png
20190522-PulserFEE1.png
Attachment 2: 20190522-PulserFEE9.png
20190522-PulserFEE9.png
Attachment 3: 20190522-PulserFEE10-Bad.png
20190522-PulserFEE10-Bad.png
Attachment 4: 20190522-PulserFEE10-VBad.png
20190522-PulserFEE10-VBad.png
Attachment 5: 20190522-Temps.png
20190522-Temps.png
Attachment 6: 20190522-Stats.png
20190522-Stats.png
Attachment 7: 20190522-StatsMerger.png
20190522-StatsMerger.png
Attachment 8: 20190522-MBSRate.png
20190522-MBSRate.png
  50   Fri Apr 12 15:15:33 2019 NHReport - FEE Kernel Panics (Update on 48)

Update on issue #48 - the "confusing state" is that the FEE has restarted and hence is undefined again.

An attached TTY log from the pi shows that the module is kernel panicking.
I have seen a couple of FEEs panic with the same error now.

(NB. The Day/time of the logs is wrong as the pi does not have the correct time - pis dont have a RTC or Internet access so the time isn't corrected)

Temperature of the modules is fine.

Aida is currently powered off (and I am away from GSI)

Quote:

it seems a FEE somtimes enters a confusing state and stops sending data
the current merger requires all FEEs to be active and so this stops the entire system from proceeding.

On MIDAS the page reports the module is "undefined"

On the TTY console (PUTTY) it returns: do_GetState returned z=0 and 8

Resetting the DAQ in question via MIDAS works (Putty logs of the stages shown) and then the merger resumes without trouble.

 

Attachment 1: aida01_log.txt
30/11:31:53|completed generic doGo
30/11:31:53|do_GetState returned z=0 and 1
30/11:31:53|get_ASICBlk : Blk=1 : bytes = 98048 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 0AE18000
30/11:31:53|Last Data : Databuffer  273188600 : 0 => 80468F1E : 1 => 00DD4000
30/11:31:53| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:53| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54|get_ASICBlk : Blk=0 : bytes = 98048 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 00DD8000
30/11:31:54|Last Data : Databuffer  273188600 : 0 => 80468F1E : 1 => 06D94000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:54| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55|get_ASICBlk : Blk=1 : bytes = 98048 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 06D98000
30/11:31:55|Last Data : Databuffer  273188600 : 0 => 80468F1E : 1 => 0CD54000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:55| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56|get_ASICBlk : Blk=0 : bytes = 98048 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 0CD58000
30/11:31:56|Last Data : Databuffer  273188600 : 0 => 80468F1F : 1 => 02D14000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:56| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57|get_ASICBlk : Blk=1 : bytes = 98048 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 02D18000
30/11:31:57|Last Data : Databuffer  273188600 : 0 => 80468F1F : 1 => 08CD4000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:57| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58|get_ASICBlk : Blk=0 : bytes = 98048 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 08CD8000
30/11:31:58|Last Data : Databuffer  273188600 : 0 => 80468F1F : 1 => 0EC94000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:58| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59|get_ASICBlk : Blk=1 : bytes = 98048 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 0EC98000
30/11:31:59|Last Data : Databuffer  273188600 : 0 => 80468F20 : 1 => 04C54000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:31:59| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00|get_ASICBlk : Blk=0 : bytes = 98048 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 04C58000
30/11:32:00|Last Data : Databuffer  273188600 : 0 => 80468F20 : 1 => 0AC14000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:00| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01|get_ASICBlk : Blk=1 : bytes = 98816 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500228 : 1 => 0AC18000
30/11:32:01|Last Data : Databuffer  273189368 : 0 => 80468F21 : 1 => 00C94000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:01| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000
30/11:32:02|do_GetState returned z=0 and 1
30/11:32:58|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
30/11:32:59|do_GetState returned z=0 and 1
30/11:33:00|Unable to handle kernel paging request for data at address 0x80501594
30/12:52:57|Faulting instruction address: 0xc0072d74
30/12:52:57|Oops: Kernel access of bad area, sig: 11 [#1]
30/12:52:57|PREEMPT Xilinx Virtex440
30/12:52:57|Modules linked in: aidamem xdriver xh_spidev_register
30/12:52:57|NIP: c0072d74 LR: c021671c CTR: c000da98
30/12:52:57|REGS: c0391cf0 TRAP: 0300   Not tainted  (2.6.31)
30/12:52:57|MSR: 00021000 <ME,CE>  CR: 22000044  XER: 20000000
30/12:52:57|DEAR: 80501594, ESR: 00000000
30/12:52:57|TASK = c036e318[0] 'swapper' THREAD: c0390000
30/12:52:57|GPR00: c021671c c0391da0 c036e318 80501594 c69352e0 00000000 0694009e 00000042
30/12:52:58|GPR08: 00000001 c69400e0 c03b8000 00000000 22000022 00005aa8 fffbbfff fff77fff
30/12:52:58|GPR16: fffff77f fffffbbf fffffffb c0390000 c0391de8 c68e13b4 c0390000 c0385dc8
30/12:52:58|GPR24: c0385bf8 c68e1000 00000055 00029000 00000042 00000002 c6935520 80501594
30/12:52:58|NIP [c0072d74] put_page+0x14/0x1a0
30/12:52:58|LR [c021671c] skb_release_data+0xac/0xd4
30/12:52:58|Call Trace:
30/12:52:58|[c0391da0] [c009191c] kfree+0x68/0xe4 (unreliable)
30/12:52:58|[c0391dc0] [c021671c] skb_release_data+0xac/0xd4
30/12:52:58|[c0391dd0] [c0216338] __kfree_skb+0x18/0xe8
30/12:52:58|[c0391de0] [c01d54a4] DmaSendHandlerBH+0x1c4/0x298
30/12:52:58|[c0391e30] [c003a968] tasklet_action+0x6c/0xec
30/12:52:58|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
30/12:52:58|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
30/12:52:58|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
30/12:52:58|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
30/12:52:58|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
30/12:52:58|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
30/12:52:58|[c0391fb0] [c000172c] rest_init+0x70/0x84
30/12:52:58|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
30/12:52:58|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
30/12:52:59|Instruction dump:
30/12:52:59|387d0140 4bfffdb9 7fe00106 4bffffb0 48235409 4bffffc0 4bffff50 9421ffe0
30/12:52:59|7c0802a6 bfa10014 90010024 7c7f1b78 <80030000> 7009c000 40820170 38030004
30/12:52:59|Kernel panic - not syncing: Fatal exception in interrupt
30/12:52:59|Call Trace:
30/12:52:59|[c0391c20] [c0005de8] show_stack+0x44/0x16c (unreliable)
30/12:52:59|[c0391c60] [c00345bc] panic+0x94/0x168
30/12:52:59|[c0391cb0] [c000bd44] die+0x178/0x18c
30/12:52:59|[c0391cd0] [c001191c] bad_page_fault+0x90/0xd8
30/12:52:59|[c0391ce0] [c000e834] handle_page_fault+0x7c/0x80
30/12:52:59|[c0391da0] [c009191c] kfree+0x68/0xe4
30/12:52:59|[c0391dc0] [c021671c] skb_release_data+0xac/0xd4
30/12:52:59|[c0391dd0] [c0216338] __kfree_skb+0x18/0xe8
30/12:52:59|[c0391de0] [c01d54a4] DmaSendHandlerBH+0x1c4/0x298
30/12:52:59|[c0391e30] [c003a968] tasklet_action+0x6c/0xec
30/12:52:59|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
30/12:52:59|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
30/12:52:59|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
30/12:52:59|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
30/12:52:59|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
30/12:52:59|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
30/12:53:00|[c0391fb0] [c000172c] rest_init+0x70/0x84
30/12:53:00|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
30/12:53:00|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
30/12:53:00|Rebooting in 180 seconds..
  49   Thu Apr 11 15:28:10 2019 CA, OH, CB, TDOffline analysis of 290319 files R13 and R16
X projection of high energy channel ADC hits from R13 (attachment 1)

XY hit pattern from R13 (attachment 2)
    -discontinuity observed
    -possible timing issues or issues with Event Clustering

X projection of high energy channel ADC hits from R16 (attachment 3)

XY hit pattern from R16 (attachment 4)
    -Beam was being diffused at this point, and tpcs were being calibrated.
    -Gap between Y strips ~ 23 -> 30

High Energy Ey vs Ex plots from R13 and R16 (attachments 5 and 6 respectively)
Attachment 1: R13highEnergyhitpattern.pdf
R13highEnergyhitpattern.pdf
Attachment 2: R13highEnergyXYhitpattern.pdf
R13highEnergyXYhitpattern.pdf
Attachment 3: R16highEnergyhitpattern.pdf
R16highEnergyhitpattern.pdf
Attachment 4: R16highEnergyXYhitpattern.pdf
R16highEnergyXYhitpattern.pdf
Attachment 5: R13HighEnergyExEy.pdf
R13HighEnergyExEy.pdf
Attachment 6: R16HighEnergyExEy.pdf
R16HighEnergyExEy.pdf
  48   Wed Apr 10 14:53:50 2019 NHReport - FEE stops sending data

it seems a FEE somtimes enters a confusing state and stops sending data
the current merger requires all FEEs to be active and so this stops the entire system from proceeding.

On MIDAS the page reports the module is "undefined"

On the TTY console (PUTTY) it returns: do_GetState returned z=0 and 8

Resetting the DAQ in question via MIDAS works (Putty logs of the stages shown) and then the merger resumes without trouble.

Attachment 1: Report-PUTTY.png
Report-PUTTY.png
Attachment 2: Report-_Merger.png
Report-_Merger.png
Attachment 3: Report_-_DAQ.png
Report_-_DAQ.png
Attachment 4: Report_-_PuTTY_Reset.png
Report_-_PuTTY_Reset.png
Attachment 5: Report_-_PUTTY_Setup.png
Report_-_PUTTY_Setup.png
Attachment 6: Report_-_PUTTY_GO.png
Report_-_PUTTY_GO.png
  47   Wed Apr 10 12:33:58 2019 NH VPAIDA Firmware Update

All AIDA modules have been updated to a new firmware 0xd330701 which should fix the merger issues.
A new GSI WR timestamp page has been added to MIDAS - the SYNC rollover is suggested to be 0xe for now.
(VP is working on lowering the rate of WR SYNC infowords)

Tested yesterday and Merger no longer reported major timestamp issues
n.b. aida11 and aida12 didn't update properly until today, this may have caused a few.

MBS relay is currently not functioning (data is confusing) but this is under investigation by VP ASAP.

 

Attachment 1: 20191004-FeeTemps.png
20191004-FeeTemps.png
Attachment 2: 20191004-GSIWR.png
20191004-GSIWR.png
  46   Thu Apr 4 14:07:45 2019 NHPulser Configuration

Pulser settings during April run:

Rate 2Hz
Delay 250 NS
Amplitude 1 V
Fall 1 ms
Polarity Pos
Pulse Top Tail
Attenuation 1X
PB5 Pulse On
Clamp Off

 

  45   Wed Apr 3 15:24:06 2019 CA, TDreport - low inquire save state
On Save/Restore Module Settings Tab - "inquire save state" returns an error. See attachment 1.
Attachment 1: 030419_1614_savestate.png
030419_1614_savestate.png
  44   Wed Apr 3 13:29:20 2019 CA, CB, TDreport - high DAQ stall
aida10 runs very fast for a period, then stops.

Have to restart Merger and TapeServer to get it running again.

As shown in attachment 1, aida10 records very high number of bad events.

Waveform for all FEE64s are disabled (see for example attachment 4) *BUT* note that 'wave events' are reported
in aida10 stats (attachment 1)
Attachment 1: 030419_1416_aida10stats.png
030419_1416_aida10stats.png
Attachment 2: 030419_1417_aida09stats.png
030419_1417_aida09stats.png
Attachment 3: 030419_1417_aida11stats.png
030419_1417_aida11stats.png
Attachment 4: 030419_1418_aida12stats.png
030419_1418_aida12stats.png
Attachment 5: 030419_1419_aida10_led.png
030419_1419_aida10_led.png
  43   Wed Apr 3 09:28:07 2019 CA, CB, TDWednesday April 3rd 2018
Arrived at 09:54 - sequenced all FEE64 on

FEE64 not mounted properly - something happened to network overnight, see attachment 1,2 for AIDA system log

10.23 - FEE64 still not mounted properly, power cycling FEE64

10.30 - sequenced all FEE64 on

10.43 - FEE64 mounted successfully

      - DAQ reset and setup - looks ok

      - histogram enabled on aida12

      - clock ReSYNC

      - system wide checks - all FEE64 pass clock status check
           
                           - aida2 and aida12 fail ADC calibration

                           - SYNC pulse and SYNC error checks ok

                           - WR decoder check returns errors (attachment 3)


10.53 - FEE64 temperatures ok (attachment 4)

      - detector bias/leakage current ok (attachment 5)

11.04 - TapeServer start - not writing to storage

      - ASIC check load performed

      - DAQ start

      - ASIC control tab (attachment 6)

      - Run control keeps reloading - has returned to 'reset' screen with aida11 showing "undefined" (attachment 7)

      - good event statistics - all FEE64 showing 0 *except* aida09 (attachment 8)

11.20 - stopped tape server

      - reset daq - returns error (attachment 9)

11.41 DAQ off, detector bias off, FEE64 off

13.30 detector bias on, FEE64 sequenced on

      all browser tabs reopened

      merger set back up - ok

      mbs data relay set back up - ok

      DAQ setup and started, initially ok

      good event statistics (attachment 10)

16.25 FEE64 restarted, along with Merger, TapeServer, DAQ (not writing to file)

      thresholds (slow comparator, fast comparator HEC, fast comparator LEC) raised to 0xff for testing.

      aida10 stats appear reasonable, however wave events still reported despite being deactivated (see report entry
on DAQ stall) (attachment 11)

17.00 DAQ, TapeServer, Merger all stopped

      Detector bias off

      FEE64s switched off

      coolant water still running (cannot be disabled due to lack of access)

      AIDA in safe state
Attachment 1: 030419_aidasystemlog.txt
Apr  3 04:06:56 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 04:06:56 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 04:06:56 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 04:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 04:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 04:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 04:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 04:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 04:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 04:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 04:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 04:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 04:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 04:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 04:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 04:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 04:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 04:54:49 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 04:54:49 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 04:54:49 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 2761 seconds.
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 04:54:49 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 05:06:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 05:06:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 05:07:22 aidas-gsi yum[25016]: Updated: tzdata-2019a-1.el6.noarch
Apr  3 05:07:23 aidas-gsi yum[25016]: Updated: tzdata-java-2019a-1.el6.noarch
Apr  3 05:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 05:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 05:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 05:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 05:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 05:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 05:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 05:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 05:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 05:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 05:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 05:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 05:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 05:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 05:40:50 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 05:40:50 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 05:40:50 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 3003 seconds.
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 05:40:50 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 06:06:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 06:06:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 06:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 06:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 06:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 06:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 06:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 06:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 06:30:53 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 06:30:53 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 06:30:53 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 3269 seconds.
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 06:30:53 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 06:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 06:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 06:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 06:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 06:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 06:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 06:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 06:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 07:06:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 07:06:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 07:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 07:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 07:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 07:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 07:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 07:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 07:25:22 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 07:25:22 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 07:25:22 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 2773 seconds.
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 07:25:22 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 07:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 07:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 07:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 07:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 07:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 07:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 07:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 07:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 08:06:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 08:06:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 08:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 08:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 08:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 08:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 08:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 08:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 08:11:35 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 08:11:35 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 08:11:35 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 3455 seconds.
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 08:11:35 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 08:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 08:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 08:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 08:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 08:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 08:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 08:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 08:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 09:06:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 09:06:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 09:08:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 09:08:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 09:08:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 09:08:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 09:08:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 09:08:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 09:09:10 aidas-gsi dhclient[3982]: DHCPREQUEST on eth0 to 10.99.0.13 port 67 (xid=0x2d1bd666)
Apr  3 09:09:10 aidas-gsi dhclient[3982]: DHCPACK from 10.99.0.13 (xid=0x2d1bd666)
Apr  3 09:09:10 aidas-gsi dhclient[3982]: bound to 10.99.2.56 -- renewal in 2865 seconds.
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info> (eth0): DHCPv4 state changed renew -> renew
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   address 10.99.2.56
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   prefix 17 (255.255.128.0)
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   gateway 10.99.0.1
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   hostname 'aida-3'
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.11'
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   nameserver '140.181.96.29'
Apr  3 09:09:10 aidas-gsi NetworkManager[3917]: <info>   domain name 'gsi.de'
Apr  3 09:36:05 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr  3 09:36:05 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr  3 09:38:00 aidas-gsi dhcpd: Dynamic and static leases present for 192.168.11.117.
Apr  3 09:38:00 aidas-gsi dhcpd: Remove host declaration nnrpi2 or remove 192.168.11.117
Apr  3 09:38:00 aidas-gsi dhcpd: from the dynamic address pool for 192.168.11.0/24
Apr  3 09:38:00 aidas-gsi dhcpd: uid lease 192.168.11.104 for client b8:27:eb:40:53:e8 is duplicate on 192.168.11.0/24
Apr  3 09:38:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr  3 09:38:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr  3 09:53:59 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.4:666 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.3:1013 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.5:904 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.8:890 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.1:1007 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.6:710 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.2:727 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.7:698 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:42:0d:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:42:0d:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:42:0d:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:42:0d:0c via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:10 via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ee:10 via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ee:10 via p4p1
Apr  3 09:54:00 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ee:10 via p4p1
Apr  3 09:54:01 aidas-gsi rpc.mountd[4324]: authenticated mount request from 192.168.11.9:907 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)














































































... 190 more lines ...
Attachment 2: 030419_usb.png
030419_usb.png
Attachment 3: WRerrorcheck.png
WRerrorcheck.png
Attachment 4: 030419_1051_temp.png
030419_1051_temp.png
Attachment 5: 030419_1052)bias.png
030419_1052)bias.png
Attachment 6: 030519_asiccontrol.png
030519_asiccontrol.png
Attachment 7: 030419_runcontrol.png
030419_runcontrol.png
Attachment 8: 030419_1116_stats.png
030419_1116_stats.png
Attachment 9: 030419_runcontrol_1120.png
030419_runcontrol_1120.png
Attachment 10: 030419_1343_stats.png
030419_1343_stats.png
Attachment 11: 030419_1652_stats10.png
030419_1652_stats10.png
  42   Tue Apr 2 09:42:20 2019 NHMBS Server setup

MBS is currently configured and accepting data:

x86l-94 : AIDA Foreign data receiver

x86l-4 : DESPEC Time sorter

The time sorter is complaining about AIDA time warps (around 1 Hz) it is assumed this is related to other merger / timestamping issues being worked on by Vic

  41   Tue Apr 2 07:39:44 2019 CB, TD, CATuesday 2nd April 2019
08:15 beam tuning in progress

08.30 System wide checks 

- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- 'collect memory information from FEE64 Linux' still returns scripting error -> will not check again until update
- WR decoder test passes
- good event statistics OK, except aida 09, 10, 11, 12 as yesterday morning (see attach 1)
- FEE temperature OK (see attach 2)
- Detector OK (see attach 3)
- NewMerger. MergerHTTPd, newMerger Terminal, data relay MBS screenshots attached (4,5,6,7)

10:37 Issues with magnets due to the new control system. Beam tuning continues

AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
- good event statistics OK, except aida 09, 10 (see attach 8). Aida 9 & 12 rates now lower
- FEE temperature OK
- Detector OK
- Merger OK

12:21 Beam start
AIDA continues running OK
- aida06 global clock failure (check clock status)
- aida02 , aida04, aida06 fail calibration (ADC calibration)
- SYNC error counter ok
- SYNC pulse check ok
- WR decoder test passes
-ASIC check load performed
-Writing to file R11 in directory 290319
-histograms zeroed
-good event statistics OK, except aida 09, 10 (attachment 9)
-FEE temperatures ok (attachment 10)
-detector bias ok (attachment 11)

-thick degrader was placed in front of snout during beam operation

12:40 beam stop
      TapeServer stopped, file R11_0 in 290319
      TapeServer restarted, not writing to file

      low energy spectra for aida09 (attachment 12)

      high energy spectra for aida09 (attachment 13)

      high energy spectra for aida11 (attachment 14)

      1D hit patterns (stat) in aida09,10,11,12 (attachment 15)

      ASIC control settings used during beam, including thresholds (attachment 16)

14.20 *NOTE degrader setting different from yesterday, silver beam still not observed, just unidentified light ions.

      AIDA continues running OK
      aida06 global clock failure (check clock status)
      aida02 , aida04, aida06 fail calibration (ADC calibration)
      SYNC error counter ok
      SYNC pulse check ok
      WR decoder test passes

14.37 good event statistics OK, except aida 10 (attachment 17)
      detector bias ok (attachment 18)
      FEE temperatures ok (attachment 19)

14.54 Degrader removed 

16:13 AIDA continues running OK
      aida06 global clock failure (check clock status)
      aida02 , aida04, aida06 fail calibration (ADC calibration)
      SYNC error counter ok
      SYNC pulse check ok
      WR decoder test passes
      FEE64 temperatures OK
      good event statistics OK, except aida 3, 9, 10 running hot (see attach 18)

17:00 Beam start - no degrader
      AIDA running OK
      Writing on file R13
      Beam is better focused (see attach 19). Appears to be off-centre in the bottom right part of the detector
      Rates OK (see attach 20)
      DSSD bias/current OK (see attach 21)

17:21 Beam re-centred. Histograms cleared - see attach 22
      AIDA continues OK
      Run continues

      High energy spectra in aida09 (attachment 25)

      Beam rate approx 5000 ions/s

      Beam being diffused
      TapeServer start stop - AIDA writing to file R14 in 290319

17:48 1D hit patterns - beam in process of being diffused (attachment 26)

17:57 leakage current back at 1.465 uA (attachment 27)

18:08 beam stopped
      TapeServer stopped and restarted - not writing to file

      updated hit patterns from beam diffusing (attachment 28)

      aida10, ADC3 high energy spectra (attachment 29)

18.11 calibrating tpc with masks

      TapeServer stop/start R16, 290319

18.49 leakage current briefly fluctuating between ~1.2 uA and ~2 uA, but has returned to stability at ~ 1.4 uA
(attachment 30)

18.55 beam stopped for tuning - TapeServer stop/start. Not writing to file.

18.58 leakage current still stable after beam off (attachment 31)

19.10 beam start. TapeServer stop/start writing to file R18 in 290319

19.15 leakage current still ok after beam activation

19.34 fee12 high energy spectra (attachment 32)

20.35 beam stop, tapeserver stopped

      histogramming link dropped on aida 12

20.38 beam start. TapeServer start R20
      
      
Attachment 1: 71.png
71.png
Attachment 2: 72.png
72.png
Attachment 3: 73.png
73.png
Attachment 4: 74.png
74.png
Attachment 5: 75.png
75.png
Attachment 6: 76.png
76.png
Attachment 7: 76.png
76.png
Attachment 8: 78.png
78.png
Attachment 9: 020419_1227_stats.png
020419_1227_stats.png
Attachment 10: 020419_1228_temp.png
020419_1228_temp.png
Attachment 11: 020419_1230_bias.png
020419_1230_bias.png
Attachment 12: 020419_1235_lowenergyspectraaida09.png
020419_1235_lowenergyspectraaida09.png
Attachment 13: 020419_HEspectra_09(2).png
020419_HEspectra_09(2).png
Attachment 14: 040219_1257_HEspectra11.png
040219_1257_HEspectra11.png
Attachment 15: highenergyhitpatters020419_1300.png
highenergyhitpatters020419_1300.png
Attachment 16: 020419_noon_asiccontrol.png
020419_noon_asiccontrol.png
Attachment 17: 020419_1434_stats.png
020419_1434_stats.png
Attachment 18: 020419_1436_bias.png
020419_1436_bias.png
Attachment 19: 040219_1454_temp.png
040219_1454_temp.png
Attachment 20: 79.png
79.png
Attachment 21: 80.png
80.png
Attachment 22: 81.png
81.png
Attachment 23: 82.png
82.png
Attachment 24: 84.png
84.png
Attachment 25: 020419_1737_HEspectra09.png
020419_1737_HEspectra09.png
Attachment 26: 020419_1748_hitpattern.png
020419_1748_hitpattern.png
Attachment 27: 020419_1756_bias.png
020419_1756_bias.png
Attachment 28: 020419_1803_hitpattern.png
020419_1803_hitpattern.png
Attachment 29: 020419_1807_315H.png
020419_1807_315H.png
Attachment 30: 020418_1851_bias.png
020418_1851_bias.png
Attachment 31: 020419_1859_bias.png
020419_1859_bias.png
Attachment 32: 040219_1932_22h_12.png
040219_1932_22h_12.png
Attachment 33: 020419_1954_hitpatter.png
020419_1954_hitpatter.png
Attachment 34: 020419_2003_hitpattern.png
020419_2003_hitpattern.png
Attachment 35: 020419_2006_waveforms.png
020419_2006_waveforms.png
Attachment 36: 020419_2035_bias.png
020419_2035_bias.png
  40   Mon Apr 1 10:20:20 2019 CA, NH, TDMBS Relay startup
To start the MBS Data Relay, click on the top rightmost of the MIDAS icons (MBS relay, Data Relay => MBS) in
Desktop 3. (attachment 1)

This will bring up the Data Relay MBS terminal.

Ensure the path defined in the datarelaymbs is set to the correct directory (shown in attachment 2) 
Attachment 1: mbsrealaypic.png
mbsrealaypic.png
Attachment 2: mbscommand.png
mbscommand.png
  39   Mon Apr 1 10:14:09 2019 CA, NH, TDFriday 29th March 2019
16:30 good event statistics ok, with 9,10,11,12 running faster (detector FEE64s) (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage currents ok (attachment 3)

      system wide checks ok *except* FEE64 Linux memory information check causes script error,
                                     WR decoder status returns errors

      DAQ & Merger running ok

      slow comparator threshold raised to 0x64

   
Attachment 1: 290319_stats.png
290319_stats.png
Attachment 2: 290319_temp.png
290319_temp.png
Attachment 3: 290913_bias.png
290913_bias.png
  38   Mon Apr 1 08:50:35 2019 CA, TDMonday April 1st 2019 - BEAM START
09.28 System wide checks - aida06 global clock failure (check clock status)

                         - aida02 , aida04, aida06 fail calibration (ADC calibration)

                         - SYNC error counter ok

                         - SYNC pulse check ok

                         - 'collect memory information from FEE64 Linux' still returns scripting error

                         - WR decoder test passes

09.33 good event statistics ok, except aida09, aida10 running hot. (attachment 1)

      FEE temperatures ok (attachment 2)

      detector bias/leakage current ok (attachment 3)

09.38 screenshots of NewMerger tab, Merger/Tape HTTPd and NewMerger terminal (attachments 4, 5, 6)

10.05 1.8.L pulser peak spectra (attachment 7)

10.13 1.8.W waveform spectra (attachment 8)

21.13 Beam on

      AIDA DAQ running - writing to file R9 in directory 290319

      System wide checks - same as above

      good event statistics ok - although rates on 9,10,11,12 have dropped (attachment 10)

      ASIC check load performed

      FEE temperatures ok (attachment 9)

      detector bias leakage currents ok (attachment 11)

      Scintillator 41 -> approx 1000 ions/s, 11 second super cycle (beam on 10s, beam off 1s)
      c. 130-150MeV/u 107Ag ions incident on plastic scintillator + AIDA DSSSD

21.36 High energy spectra for aida09 - (attachment 12)

      High energy spectra for aida11 - (attachment 13)

      Low energy spectra - (attachment 14)

      1D hit pattern - (attachment 15)

21.53 ASIC control settings - see for slow comparator threshold, fast comparator threshold (HEC) and fast
      comparator threshold (LEC) (attachment 16)

      NewMerger screenshot - working (attachment 17)

22.00 ASIC options - including directory path to saved ASIC settings (attachment 18)

22.09 beam stop
      TapeServer stopped, restarted not writing to storage
      one complete file produced R9_0 (2.0 GB)
      plus R9_1 (154 MB)

*NOTE FROM 02/04/19 - DEGRADER in front of detector, silver beam not observed, just unidentified light ions.
Attachment 1: 010419_0930_stats.png
010419_0930_stats.png
Attachment 2: 010419_0933_temp.png
010419_0933_temp.png
Attachment 3: 010419_0933_bias.png
010419_0933_bias.png
Attachment 4: 010419_0936_mergertab.png
010419_0936_mergertab.png
Attachment 5: 010419_0937_mergerterm.png
010419_0937_mergerterm.png
Attachment 6: 010419_0938_newmergerterm.png
010419_0938_newmergerterm.png
Attachment 7: 010419_1005_18L.png
010419_1005_18L.png
Attachment 8: 101419_1013_18W.png
101419_1013_18W.png
Attachment 9: 010419_2108_temp.png
010419_2108_temp.png
Attachment 10: 010419_2109_stats.png
010419_2109_stats.png
Attachment 11: 010419_2111_bias.png
010419_2111_bias.png
Attachment 12: 010419_2136_115Haida09.png
010419_2136_115Haida09.png
Attachment 13: 010419_2139_1H_aida11.png
010419_2139_1H_aida11.png
Attachment 14: 010419_lowenergyspectra.png
010419_lowenergyspectra.png
Attachment 15: 010419_1Dhitpatternstat.png
010419_1Dhitpatternstat.png
Attachment 16: 010419_AsicSettings.png
010419_AsicSettings.png
Attachment 17: 010419_newmerger_2150.png
010419_newmerger_2150.png
Attachment 18: 010419_asicsettings.png
010419_asicsettings.png
  37   Tue Mar 26 14:59:30 2019 NH26.03.2019

Fig1: FEE temperatures from 26.03.2019  all OK

AIDA has been on today and merger setting fixed (see previous ELOG entry https://elog.ph.ed.ac.uk/DESPEC/36)

Still diagnosing issues with MBS (figure 2) but local data acquisition should be functional.

Detector has not been biased.

Attachment 1: 20190326Temps.png
20190326Temps.png
Attachment 2: aidambsrelay-error.png
aidambsrelay-error.png
  36   Tue Mar 26 14:58:09 2019 NH, VPHow to reset merger

If the merger doesn't show "xfer Links => Merger" at the top, it is probably the merger setting has been corrupted somehow. This is how to reset it

  If you go to the web page from which you started the Merge and TapeServer

    (probably    localhost:8115)

 Then select the item  NetVar Service  which is at the left hand end.

  Click on Inquire Registers

  then  enter    NetVar.MERGE.RunOptions  into the "Register Name" entry    and  when finished move the mouse out of the entry box.

 NetVar.MERGE.RunOptions will appear in the center of the top row.   (it is an item in the menu but not so easy to find)

 Click on Read and I expect you will get  0

 Type 1 into the data entry box     and click on Write

  Now return to the Merger and  click on  Reload

  The Merger State will now end as    xfer Links => Merger

 The button  Output:   will now toggle between this and   xfer links => Merger => Storage.

  35   Mon Mar 25 14:44:33 2019 NHMerger issues

Plastic electronics have now been installed.

Water is not leaking still.

FEE64s powered on at 15:00 to test MBS connection (fig1: temperatures)

MBS relay connects to MBS server OK...

AIDA Merger does not accept any data... (links with data reads 0) and lots of timestamp errors in log? (Figs 2-4)


Startup procedure incorrect?

Otherwise things still OK.

Attachment 1: Temps.png
Temps.png
Attachment 2: AIDARUNv2.png
AIDARUNv2.png
Attachment 3: Mergerv2.png
Mergerv2.png
Attachment 4: Tapev2.png
Tapev2.png
Attachment 5: MergerConsolev2.png
MergerConsolev2.png
  34   Fri Mar 22 15:44:20 2019 NHEnd of 22.03.2019

End of the day - Detector unbiased and FEE64s turned off for the weekend as they won't be monitored.

Water will remain on.

  33   Fri Mar 22 14:31:24 2019 NHA few more setup notes I forgot

Extra FEE information:

HV braid (0 V) goes into FEE10 (beam left) with LK1 jumper
HV core (-160 V) goes into FEE09 (top)

Could be related to noise in FEE09?
Both these FEEs have higher rates too

Other setup information:
According to the switch in S4, the AIDA workstation is only connected via 100Mbps rather than 1Gbps. (Fig 1)

Plastic is inside the snout, but is not yet connected to the electronics yet
Figures 2-4 Show the DSSD & Plastic in the Snout. Plastic is far upstream solely to ensure plenty of ribbon cable

Attachment 1: 20190322_132528s.jpg
20190322_132528s.jpg
Attachment 2: 20190319_131821s.jpg
20190319_131821s.jpg
Attachment 3: 20190313_161102s.jpg
20190313_161102s.jpg
Attachment 4: 20190313_155418s.jpg
20190313_155418s.jpg
  32   Fri Mar 22 13:47:32 2019 NHResolution Checks

Pulser widths of AIDA modules, resolution is fairly typical. AIDA09 still noisier than other DSSD channels.

Figure 1: Picture of pulser peaks
Figure 2: Raw waves (AIDA01 did not calibrate, so AIDA02 used instead as "non DSSD module")
Figure 3: Good Events

Fast comparator threshold (LEC/MEC) changed to 0xFF

AIDA09 seems noiser than before(?) but is known to be noisy.

AIDA01:  24.22

AIDA09: 205.21

AIDA10: 88.60

AIDA11: 63.33

AIDA12: 102.69

Attachment 1: 20190322Resolutions.png
20190322Resolutions.png
Attachment 2: 20190323Waves.png
20190323Waves.png
Attachment 3: 20190322Rates.png
20190322Rates.png
  31   Fri Mar 22 13:32:55 2019 NHAIDA Startup @ 22.03.2019

FEEs have been on this morning, temperatures attached

Leakage current attached

the DSSD is attached to the following AIDA modules:
Beam Left: 10
Beam Right: 12
Bottom: 11
Top: 9

MIDAS Servers on, will be looking at resolution (pulser)

Attachment 1: 20190323Bias.png
20190323Bias.png
Attachment 2: 20190322FEETemps.png
20190322FEETemps.png
  30   Thu Mar 21 15:45:48 2019 NH, AS, SA, MPSnout Mounting & DSSD Bias

The AIDA snout (with single DSSD and plastic) has been mounted in S4 cave again. DSSD was connected to the outer 4 FEEs again.

-160 V Bias applied and DSSD holds voltage with leakage current of around 1.3 uA (was dropping as only biased briefly)

Bias off and FEEs offline again now as day nearly done. MBS testing will be performed shortly, as well as resolution checks.

Attachment 1: LeakageBias.png
LeakageBias.png
  29   Thu Mar 21 14:10:10 2019 NHFEE Temperatures

All 12 FEEs powered up, and temperatures are OK.

Passed ASIC checks and system wide checks - now checking cabling and to mount snout.

Attachment 1: FEETemps.png
FEETemps.png
  28   Thu Mar 21 13:59:08 2019 NHAIDA Water

The AIDA water system has been repaired and not leaking.

The interlock did not light immediately, but power cable reconnected to box and now it works.

 

  27   Wed Mar 20 15:35:10 2019 NHAIDA Setup @ GSI

Currently

- 1 DSSD and Plastic is in the AIDA snout, which is not mounted

- The FEEs are offline due to a water supply issue, this will be repaired early 21/03/2019

- The pis & aida-3 are online

The water interlock also didn't clear, which will be investigated

Ideally installation can be completed tomorrow but this depends on water

  26   Sun Feb 3 16:01:49 2019 TDUpdated analyses GSIJan19/R1_0 - R10_0
Summary statistics for WR test data files R1_0, R2_0, R3_0, R4_0, R7_0 and R10_0 (attachments 1-6)

Latest version of WR TDR data analyser (attachment 7)
Attachment 1: R1_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 ***   SYNC100 timestamp: block:  1894 ptr: 13041 data: 0x874835E5 module:  7 information type:  4 information field: 0x000835E5 ts: 0x157C835E5A000000 OLD: ts: 0x157C835E7A000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  1895 ptr:  7879 data: 0xC1D48169 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33129  ts: 0x157C835E58849AFE OLD: ts: 0x157C835E5C3F1E1E dt: 0x0000000003BA8320 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1943 ptr:  5113 data: 0x874835E8 module:  7 information type:  4 information field: 0x000835E8 ts: 0x157C835E800002BE OLD: ts: 0x157C835F500002BE dt: 0x00000000D0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1944 ptr: 12411 data: 0x874835E6 module:  7 information type:  4 information field: 0x000835E6 ts: 0x157C835E60001DBE OLD: ts: 0x157C835E80001DBE dt: 0x0000000020000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1946 ptr:  4475 data: 0x874835D8 module:  7 information type:  4 information field: 0x000835D8 ts: 0x157C835D88000000 OLD: ts: 0x157C835E68000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***            ADC data: block:  1947 ptr:  3175 data: 0xC1FD8258 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33368  ts: 0x157C835D8B65444E OLD: ts: 0x157C835D8C426C9E dt: 0x0000000000DD2850 tag: 3 type: 0
 ***            ADC data: block:  1957 ptr:  6665 data: 0xC1C5823C module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33340  ts: 0x157C835F89CC0C2E OLD: ts: 0x157C835F89F902C4 dt: 0x00000000002CF696 tag: 2 type: 6
 ***            ADC data: block:  1957 ptr: 12881 data: 0xC1FD80EC module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33004  ts: 0x157C835F8A7E5A7E OLD: ts: 0x157C835F8B071C6E dt: 0x000000000088C1F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1958 ptr:  3705 data: 0x874835F7 module:  7 information type:  4 information field: 0x000835F7 ts: 0x157C835F7C000000 OLD: ts: 0x157C835F8C000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1958 ptr: 10857 data: 0xC1E6808B module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32907  ts: 0x157C835F7B071C6E OLD: ts: 0x157C835F7DCBC05E dt: 0x0000000002C4A3F0 tag: 3 type: 0
 ***            ADC data: block:  1960 ptr:  7503 data: 0xC1D48126 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33062  ts: 0x157C835FA31908DE OLD: ts: 0x157C835FA738E804 dt: 0x00000000041FDF26 tag: 2 type: 3
 ***   SYNC100 timestamp: block:  1961 ptr: 12821 data: 0x874835F6 module:  7 information type:  4 information field: 0x000835F6 ts: 0x157C835F68000000 OLD: ts: 0x157C835FA8000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  1962 ptr:  3459 data: 0xC1E3814D module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33101  ts: 0x157C835F697AF9EE OLD: ts: 0x157C835F6B54564E dt: 0x0000000001D95C60 tag: 3 type: 0
 ***            ADC data: block:  1993 ptr: 14241 data: 0xC1EF8096 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32918  ts: 0x157C836017EE3F5E OLD: ts: 0x157C836018F24D14 dt: 0x0000000001040DB6 tag: 2 type: 6
 ***            ADC data: block:  1994 ptr: 11543 data: 0xC1EC8304 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33540  ts: 0x157C8360155457CE OLD: ts: 0x157C83601ABE10CE dt: 0x000000000569B900 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1995 ptr: 15735 data: 0x874835FB module:  7 information type:  4 information field: 0x000835FB ts: 0x157C835FB000040E OLD: ts: 0x157C83601000040E dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  2005 ptr: 12983 data: 0xC1FF8214 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33300  ts: 0x157C836072C1877E OLD: ts: 0x157C8360735F143A dt: 0x00000000009D8CBC tag: 2 type: 6
 ***            ADC data: block:  2007 ptr:  7253 data: 0xC1FAC5BF module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 50623  ts: 0x157C83607AEEAECE OLD: ts: 0x157C83607B19629E dt: 0x00000000002AB3D0 tag: 3 type: 0
 ***            ADC data: block:  2008 ptr:  5229 data: 0xC1C580F3 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33011  ts: 0x157C8360754432EE OLD: ts: 0x157C83607E4EC1EE dt: 0x00000000090A8F00 tag: 3 type: 0
 ***            ADC data: block:  2008 ptr:  9407 data: 0xC1F181AC module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33196  ts: 0x157C836075F4684E OLD: ts: 0x157C836076000000 dt: 0x00000000000B97B2 tag: 2 type: 4
 ***            ADC data: block:  2026 ptr: 12273 data: 0xC1D08030 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32816  ts: 0x157C8361083A8ECE OLD: ts: 0x157C83610CD0576E dt: 0x000000000495C8A0 tag: 3 type: 0
 ***            ADC data: block:  2031 ptr: 14603 data: 0xC1EF80B0 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32944  ts: 0x157C8361138087BE OLD: ts: 0x157C83611D490BFE dt: 0x0000000009C88440 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2033 ptr: 10651 data: 0x87483606 module:  7 information type:  4 information field: 0x00083606 ts: 0x157C8360600470CE OLD: ts: 0x157C8360E00470CE dt: 0x0000000080000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  2034 ptr: 14335 data: 0x874835FB module:  7 information type:  4 information field: 0x000835FB ts: 0x157C835FB8000000 OLD: ts: 0x157C836068000000 dt: 0x00000000B0000000 tag: 2 type: 5
 ***            ADC data: block:  2036 ptr: 11667 data: 0xC1EF8157 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33111  ts: 0x157C835FB369CADE OLD: ts: 0x157C835FBEF53D4E dt: 0x000000000B8B7270 tag: 3 type: 0
 ***            ADC data: block:  2036 ptr: 13715 data: 0xC1DC8081 module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32897  ts: 0x157C835FB127EFEE OLD: ts: 0x157C835FB3F282DE dt: 0x0000000002CA92F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2037 ptr:  1405 data: 0x874835D9 module:  7 information type:  4 information field: 0x000835D9 ts: 0x157C835D92000000 OLD: ts: 0x157C835FB2000000 dt: 0x0000000220000000 tag: 2 type: 5
 ***            ADC data: block:  2038 ptr: 11415 data: 0xC1E0816D module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33133  ts: 0x157C835D97863F4E OLD: ts: 0x157C835D9E71BF30 dt: 0x0000000006EB7FE2 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2061 ptr: 11073 data: 0x8748361E module:  7 information type:  4 information field: 0x0008361E ts: 0x157C8361E000004E OLD: ts: 0x157C83621000004E dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  2112 ptr:  3409 data: 0xC1CA80A9 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32937  ts: 0x157C8362BDBB46BE OLD: ts: 0x157C8362BDE9BBA2 dt: 0x00000000002E74E4 tag: 2 type: 6
 ***            ADC data: block:  2112 ptr:  8397 data: 0xC1FF8195 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33173  ts: 0x157C8362B25850AE OLD: ts: 0x157C8362BEA0D45E dt: 0x000000000C4883B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2112 ptr: 16041 data: 0x87483622 module:  7 information type:  4 information field: 0x00083622 ts: 0x157C836224000000 OLD: ts: 0x157C8362B4000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  2118 ptr:  2881 data: 0xC1E180FE module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33022  ts: 0x157C8363203CE23E OLD: ts: 0x157C83632748CA8E dt: 0x00000000070BE850 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2120 ptr:  5839 data: 0x8748361E module:  7 information type:  4 information field: 0x0008361E ts: 0x157C8361E8000000 OLD: ts: 0x157C836328000000 dt: 0x0000000140000000 tag: 2 type: 5
 ***            ADC data: block:  2132 ptr:  6753 data: 0xC1FC80AC module:  7 fail: 0 range: 0 id:   508 channel: 60 adc: 32940  ts: 0x157C836357CD4AFE OLD: ts: 0x157C836357DE0556 dt: 0x000000000010BA58 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2134 ptr:  8551 data: 0x87483633 module:  7 information type:  4 information field: 0x00083633 ts: 0x157C8363300005DE OLD: ts: 0x157C8363600005DE dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  2138 ptr:  5621 data: 0xC1CA80C1 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32961  ts: 0x157C836371ABB2EE OLD: ts: 0x157C836372BFFEEC dt: 0x0000000001144BFE tag: 2 type: 6
 ***            ADC data: block:  2139 ptr:  7423 data: 0xC1DB81E9 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33257  ts: 0x157C83637295AD5E OLD: ts: 0x157C83637C25164E dt: 0x00000000098F68F0 tag: 3 type: 0
 ***            ADC data: block:  2148 ptr:  9879 data: 0xC1F7830F module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33551  ts: 0x157C8363AEE1D59E OLD: ts: 0x157C8363AFD7C7BA dt: 0x0000000000F5F21C tag: 2 type: 6
 ***            ADC data: block:  2150 ptr:   511 data: 0xC1C58121 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33057  ts: 0x157C8363B1144C6E OLD: ts: 0x157C8363B39F2CAE dt: 0x00000000028AE040 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2150 ptr:  5131 data: 0x87483633 module:  7 information type:  4 information field: 0x00083633 ts: 0x157C836332000000 OLD: ts: 0x157C8363B2000000 dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block:  2150 ptr: 14859 data: 0xC1D07F79 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32633  ts: 0x157C83633083AE5E OLD: ts: 0x157C83633403948E dt: 0x00000000037FE630 tag: 3 type: 0
 ***            ADC data: block:  2174 ptr: 15085 data: 0xC1DA814C module:  7 fail: 0 range: 0 id:   474 channel: 26 adc: 33100  ts: 0x157C836463324BBE OLD: ts: 0x157C83646378E470 dt: 0x00000000004698B2 tag: 2 type: 6
 ***            ADC data: block:  2194 ptr:  3783 data: 0xC1F48236 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33334  ts: 0x157C8364A6768A8E OLD: ts: 0x157C8364A6AE8AB0 dt: 0x0000000000380022 tag: 2 type: 6
 ***            ADC data: block:  2195 ptr: 15621 data: 0xC1EA807A module:  7 fail: 0 range: 0 id:   490 channel: 42 adc: 32890  ts: 0x157C8364A2B76DEE OLD: ts: 0x157C8364AF58DBBE dt: 0x000000000CA16DD0 tag: 3 type: 0
 ***            ADC data: block:  2203 ptr:  7109 data: 0xC1C58129 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33065  ts: 0x157C8365307A080E OLD: ts: 0x157C83653F701B2E dt: 0x000000000EF61320 tag: 3 type: 0
 ***            ADC data: block:  2209 ptr:  6391 data: 0xC1D38101 module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 33025  ts: 0x157C836550D977DE OLD: ts: 0x157C83655D1FE85E dt: 0x000000000C467080 tag: 3 type: 0
 ***            ADC data: block:  2216 ptr:  7037 data: 0xC1FA8130 module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 33072  ts: 0x157C83657E2FD04E OLD: ts: 0x157C83657EFC984E dt: 0x0000000000CCC800 tag: 3 type: 0
 ***            ADC data: block:  2264 ptr: 11897 data: 0xC1C580AC module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32940  ts: 0x157C836617E4E1CE OLD: ts: 0x157C836618F0C722 dt: 0x00000000010BE554 tag: 2 type: 6
 ***            ADC data: block:  2265 ptr:  2501 data: 0xC1F480A3 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32931  ts: 0x157C836616116E3E OLD: ts: 0x157C836619579C0E dt: 0x0000000003462DD0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2266 ptr: 12885 data: 0x87483654 module:  7 information type:  4 information field: 0x00083654 ts: 0x157C83654001A3BE OLD: ts: 0x157C83661001A3BE dt: 0x00000000D0000000 tag: 2 type: 5
 ***            ADC data: block:  2277 ptr:  5309 data: 0xC1C780C5 module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32965  ts: 0x157C836685CDA96E OLD: ts: 0x157C836686EAD1DC dt: 0x00000000011D286E tag: 2 type: 6
 ***            ADC data: block:  2278 ptr:  2237 data: 0xC1C0819E module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33182  ts: 0x157C8366886042FE OLD: ts: 0x157C8366895C7D4E dt: 0x0000000000FC3A50 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2278 ptr:  2741 data: 0x87483667 module:  7 information type:  4 information field: 0x00083667 ts: 0x157C836670000000 OLD: ts: 0x157C836680000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  2280 ptr:  5235 data: 0xC1F78087 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32903  ts: 0x157C836678882DBE OLD: ts: 0x157C83667AA0C1AE dt: 0x00000000021893F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2296 ptr:  1483 data: 0x8748366F module:  7 information type:  4 information field: 0x0008366F ts: 0x157C8366F2000000 OLD: ts: 0x157C836712000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  2334 ptr: 13797 data: 0xC1F3801C module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32796  ts: 0x157C83679D7CB0FE OLD: ts: 0x157C83679D84430A dt: 0x000000000007920C tag: 2 type: 6
 ***            ADC data: block:  2335 ptr:  6511 data: 0xC1CD8171 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33137  ts: 0x157C8367916B5E8E OLD: ts: 0x157C83679EE4D44E dt: 0x000000000D7975C0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2336 ptr:  2869 data: 0x87483672 module:  7 information type:  4 information field: 0x00083672 ts: 0x157C836724000000 OLD: ts: 0x157C836794000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block:  2338 ptr:   443 data: 0xC1D280B4 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32948  ts: 0x157C83672621CBBE OLD: ts: 0x157C83672A67539E dt: 0x00000000044587E0 tag: 3 type: 0
 ***            ADC data: block:  2343 ptr:  5131 data: 0xC1F281D8 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33240  ts: 0x157C8367E408D76E OLD: ts: 0x157C8367EAC7914E dt: 0x0000000006BEB9E0 tag: 3 type: 0
 *** FEE64 discriminator: block:  2359 ptr: 12299 data: 0x84640010 module:  4 information type:  6 information field: 0x00040010 ts: 0x157C83686991DD68 OLD: ts: 0x157C83686EA8B37E dt: 0x000000000516D616 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2360 ptr:  1683 data: 0x87483684 module:  7 information type:  4 information field: 0x00083684 ts: 0x157C836840000DEE OLD: ts: 0x157C836860000DEE dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  2361 ptr:  6885 data: 0xC1D58026 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32806  ts: 0x157C8368411F415E OLD: ts: 0x157C83684C0D370E dt: 0x000000000AEDF5B0 tag: 3 type: 0
 ***            ADC data: block:  2363 ptr:  6841 data: 0xC1F28236 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33334  ts: 0x157C8368704D637E OLD: ts: 0x157C83687F94C31E dt: 0x000000000F475FA0 tag: 3 type: 0
 ***            ADC data: block:  2389 ptr:  3211 data: 0xC1C87FD7 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32727  ts: 0x157C8368D314A46E OLD: ts: 0x157C8368D3E15BE4 dt: 0x0000000000CCB776 tag: 2 type: 6
 ***            ADC data: block:  2390 ptr:  6445 data: 0xC1C87FF1 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32753  ts: 0x157C8368D555459E OLD: ts: 0x157C8368D9D9422E dt: 0x000000000483FC90 tag: 3 type: 0
 ***            ADC data: block:  2427 ptr:  4063 data: 0xC1DA80CB module:  7 fail: 0 range: 0 id:   474 channel: 26 adc: 32971  ts: 0x157C836980E7C4BE OLD: ts: 0x157C83698BC2A84E dt: 0x000000000ADAE390 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2427 ptr:  8199 data: 0x87483692 module:  7 information type:  4 information field: 0x00083692 ts: 0x157C836922000000 OLD: ts: 0x157C836982000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  2429 ptr: 12991 data: 0x87483688 module:  7 information type:  4 information field: 0x00083688 ts: 0x157C836880041DAE OLD: ts: 0x157C836920041DAE dt: 0x00000000A0000000 tag: 2 type: 5
 ***            ADC data: block:  2430 ptr: 12345 data: 0xC1EB8140 module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33088  ts: 0x157C83688BC2B01E OLD: ts: 0x157C83688DAB3AFE dt: 0x0000000001E88AE0 tag: 3 type: 0
 ***            ADC data: block:  2433 ptr:  1119 data: 0xC1F78077 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32887  ts: 0x157C8369AC0A6BBE OLD: ts: 0x157C8369AC5BB28E dt: 0x00000000005146D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2434 ptr:  1961 data: 0x8748361F module:  7 information type:  4 information field: 0x0008361F ts: 0x157C8361F0006FCE OLD: ts: 0x157C8369A0006FCE dt: 0x00000007B0000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  2436 ptr:  9407 data: 0x88640010 module:  8 information type:  6 information field: 0x00040010 ts: 0x157C8361F742676C OLD: ts: 0x157C8361F853C0FE dt: 0x0000000001115992 tag: 3 type: 0
 ***            ADC data: block:  2437 ptr: 14417 data: 0xC1EC8194 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33172  ts: 0x157C8369B1470EDE OLD: ts: 0x157C8369B241B5AA dt: 0x0000000000FAA6CC tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2458 ptr:  8113 data: 0x874836A2 module:  7 information type:  4 information field: 0x000836A2 ts: 0x157C836A26000000 OLD: ts: 0x157C836A56000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  2459 ptr:  5987 data: 0xC1E1820F module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33295  ts: 0x157C836A20FEB90E OLD: ts: 0x157C836A2918275E dt: 0x0000000008196E50 tag: 3 type: 0
 ***            ADC data: block:  2485 ptr: 13361 data: 0xC1CD808A module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 32906  ts: 0x157C836AF8BEB7BE OLD: ts: 0x157C836AF9096E1E dt: 0x00000000004AB660 tag: 3 type: 0
 ***            ADC data: block:  2487 ptr:  9205 data: 0xC1C87FB3 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32691  ts: 0x157C836A99097DBE OLD: ts: 0x157C836A9FB8A46E dt: 0x0000000006AF26B0 tag: 3 type: 0
 ***            ADC data: block:  2503 ptr:  4059 data: 0xC1C5817A module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33146  ts: 0x157C836B3250F17E OLD: ts: 0x157C836B332CC1D6 dt: 0x0000000000DBD058 tag: 2 type: 6
 ***            ADC data: block:  2511 ptr:   683 data: 0xC1FC815A module:  7 fail: 0 range: 0 id:   508 channel: 60 adc: 33114  ts: 0x157C836B9FF2098E OLD: ts: 0x157C836B9FF2356C dt: 0x0000000000002BDE tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2511 ptr:  4789 data: 0x874836B0 module:  7 information type:  4 information field: 0x000836B0 ts: 0x157C836B0000157E OLD: ts: 0x157C836BA000157E dt: 0x00000000A0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  2512 ptr:  5617 data: 0x874836A7 module:  7 information type:  4 information field: 0x000836A7 ts: 0x157C836A700048AE OLD: ts: 0x157C836B000048AE dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  2512 ptr: 16097 data: 0xC1C18149 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33097  ts: 0x157C836A70B8284E OLD: ts: 0x157C836A7251A31E dt: 0x0000000001997AD0 tag: 3 type: 0
 *** FEE64 discriminator: block:  2517 ptr:  4645 data: 0x88640010 module:  8 information type:  6 information field: 0x00040010 ts: 0x157C836BF44056F8 OLD: ts: 0x157C836BFA4B66BE dt: 0x00000000060B0FC6 tag: 3 type: 0
 ***            ADC data: block:  2567 ptr: 11167 data: 0xC1E080C1 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 32961  ts: 0x157C836C9DF2A26E OLD: ts: 0x157C836C9E18F2C0 dt: 0x0000000000265052 tag: 2 type: 6
 ***            ADC data: block:  2568 ptr:  4677 data: 0xC1E980DC module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32988  ts: 0x157C836C93A1F71E OLD: ts: 0x157C836C9FE6832E dt: 0x000000000C448C10 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2569 ptr:    43 data: 0x874836C0 module:  7 information type:  4 information field: 0x000836C0 ts: 0x157C836C06000000 OLD: ts: 0x157C836C96000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  2584 ptr:  3545 data: 0xC1D380DA module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 32986  ts: 0x157C836D0666D69E OLD: ts: 0x157C836D072FE854 dt: 0x0000000000C911B6 tag: 2 type: 6
 ***            ADC data: block:  2587 ptr:  8351 data: 0xC1CD8139 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33081  ts: 0x157C836D30016E7E OLD: ts: 0x157C836D38A84FAC dt: 0x0000000008A6E12E tag: 2 type: 3
 ***   SYNC100 timestamp: block:  2588 ptr:  1083 data: 0x874836CE module:  7 information type:  4 information field: 0x000836CE ts: 0x157C836CE2000000 OLD: ts: 0x157C836D32000000 dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block:  2608 ptr:  9771 data: 0xC1E68072 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32882  ts: 0x157C836D80B1C0EE OLD: ts: 0x157C836D816340F8 dt: 0x0000000000B1800A tag: 2 type: 6
 ***            ADC data: block:  2630 ptr:  2687 data: 0xC1E6805D module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32861  ts: 0x157C836DFE37543E OLD: ts: 0x157C836DFF400F38 dt: 0x000000000108BAFA tag: 2 type: 6
 ***            ADC data: block:  2630 ptr:  6351 data: 0xC1E98060 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32864  ts: 0x157C836DF22D735E OLD: ts: 0x157C836DFEFCA7DE dt: 0x000000000CCF3480 tag: 3 type: 0
 ***            ADC data: block:  2635 ptr:  9907 data: 0xC1D48199 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33177  ts: 0x157C836E33E65EFE OLD: ts: 0x157C836E3504809A dt: 0x00000000011E219C tag: 2 type: 6
 ***            ADC data: block:  2654 ptr:  7437 data: 0xC1C78154 module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 33108  ts: 0x157C836E75E5C98E OLD: ts: 0x157C836E77000D3E dt: 0x00000000011A43B0 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2670 ptr: 15405 data: 0x874836ED module:  7 information type:  4 information field: 0x000836ED ts: 0x157C836ED00050CE OLD: ts: 0x157C836EE00050CE dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  2688 ptr: 15583 data: 0xC1FF8242 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33346  ts: 0x157C836F2190FDEE OLD: ts: 0x157C836F220F9C26 dt: 0x00000000007E9E38 tag: 2 type: 6
 ***            ADC data: block:  2690 ptr:  9131 data: 0xC1F78273 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33395  ts: 0x157C836F23D2B85E OLD: ts: 0x157C836F2F7AC98E dt: 0x000000000BA81130 tag: 3 type: 0
 ***            ADC data: block:  2708 ptr:  4285 data: 0xC1F780C7 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32967  ts: 0x157C836F79128ACE OLD: ts: 0x157C836F7A08940A dt: 0x0000000000F6093C tag: 2 type: 6
 ***            ADC data: block:  2708 ptr: 12267 data: 0xC1F48064 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32868  ts: 0x157C836F76C57F5E OLD: ts: 0x157C836F7AD6C83E dt: 0x00000000041148E0 tag: 3 type: 0
 ***            ADC data: block:  2716 ptr:   131 data: 0xC1DA812B module:  7 fail: 0 range: 0 id:   474 channel: 26 adc: 33067  ts: 0x157C836FC102B5DE OLD: ts: 0x157C836FC7AFF82E dt: 0x0000000006AD4250 tag: 3 type: 0
 ***            ADC data: block:  2729 ptr:  5645 data: 0xC1E0815B module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33115  ts: 0x157C836FF33841AE OLD: ts: 0x157C836FF43B12D4 dt: 0x000000000102D126 tag: 2 type: 6
 *** FEE64 discriminator: block:  2731 ptr:  2853 data: 0x88640010 module:  8 information type:  6 information field: 0x00040010 ts: 0x157C836FF6ACC238 OLD: ts: 0x157C836FFB09F0CE dt: 0x00000000045D2E96 tag: 3 type: 0
 ***            ADC data: block:  2736 ptr: 13505 data: 0xC1E081AA module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33194  ts: 0x157C8370146B66EE OLD: ts: 0x157C837015391238 dt: 0x0000000000CDAB4A tag: 2 type: 6
 ***            ADC data: block:  2737 ptr:  8361 data: 0xC1C5815D module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33117  ts: 0x157C8370104BC4BE OLD: ts: 0x157C8370167B535E dt: 0x00000000062F8EA0 tag: 3 type: 0
 ***            ADC data: block:  2743 ptr:  7103 data: 0xC1E081B2 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33202  ts: 0x157C837051C82E5E OLD: ts: 0x157C83705E838AAE dt: 0x000000000CBB5C50 tag: 3 type: 0
 ***            ADC data: block:  2756 ptr:  7363 data: 0xC1F37FE7 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32743  ts: 0x157C83708D1D2B0E OLD: ts: 0x157C83708E000000 dt: 0x0000000000E2D4F2 tag: 2 type: 4
 ***            ADC data: block:  2792 ptr:  4217 data: 0xC1FF8291 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33425  ts: 0x157C83710C08034E OLD: ts: 0x157C83710C0E5712 dt: 0x00000000000653C4 tag: 2 type: 6
 ***            ADC data: block:  2792 ptr: 14227 data: 0xC1DB8285 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33413  ts: 0x157C83710B1289BE OLD: ts: 0x157C83710E048ECE dt: 0x0000000002F20510 tag: 3 type: 0
 ***            ADC data: block:  2793 ptr: 12203 data: 0xC1F28126 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33062  ts: 0x157C8371057B3ECE OLD: ts: 0x157C83710E6CB33C dt: 0x0000000008F1746E tag: 2 type: 6
 ***   SYNC100 timestamp: block:  2793 ptr: 14767 data: 0x874836D4 module:  7 information type:  4 information field: 0x000836D4 ts: 0x157C836D46000000 OLD: ts: 0x157C837106000000 dt: 0x00000003C0000000 tag: 2 type: 5
 ***            ADC data: block:  2812 ptr:  4953 data: 0xC1F480F0 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33008  ts: 0x157C83717000F99E OLD: ts: 0x157C837174EEE6DE dt: 0x0000000004EDED40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2812 ptr: 13831 data: 0x87483714 module:  7 information type:  4 information field: 0x00083714 ts: 0x157C837142000000 OLD: ts: 0x157C837172000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  2815 ptr:  2669 data: 0xC1C581AC module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33196  ts: 0x157C837193558B6E OLD: ts: 0x157C83719CEBEA1A dt: 0x0000000009965EAC tag: 2 type: 3
 ***            ADC data: block:  2829 ptr: 13327 data: 0xC1E381D6 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33238  ts: 0x157C8371D4F2666E OLD: ts: 0x157C8371D5AAEAEA dt: 0x0000000000B8847C tag: 2 type: 6
 ***            ADC data: block:  2830 ptr:  9809 data: 0xC1CD8230 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33328  ts: 0x157C8371D259D30E OLD: ts: 0x157C8371D764700E dt: 0x00000000050A9D00 tag: 3 type: 0
 ***            ADC data: block:  2832 ptr:  6873 data: 0xC1C081CC module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33228  ts: 0x157C8371E5A51DDE OLD: ts: 0x157C8371EA4E7A8E dt: 0x0000000004A95CB0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2832 ptr:  8259 data: 0x8748371A module:  7 information type:  4 information field: 0x0008371A ts: 0x157C8371A6000000 OLD: ts: 0x157C8371E6000000 dt: 0x0000000040000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  2834 ptr:  2827 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157C8371A31BF408 OLD: ts: 0x157C8371AF77279E dt: 0x000000000C5B3396 tag: 3 type: 0
 ***            ADC data: block:  2855 ptr:  1439 data: 0xC1C87FED module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32749  ts: 0x157C83727F3A2A9E OLD: ts: 0x157C83727F835688 dt: 0x0000000000492BEA tag: 2 type: 6
 *** FEE64 discriminator: block:  2885 ptr: 10459 data: 0x84641010 module:  4 information type:  6 information field: 0x00041010 ts: 0x157C8372E27D2D36 OLD: ts: 0x157C8372E53404AE dt: 0x0000000002B6D778 tag: 3 type: 0
 ***            ADC data: block:  2886 ptr:  2269 data: 0xC1C58256 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33366  ts: 0x157C8372E1A93B8E OLD: ts: 0x157C8372E6C1EFDE dt: 0x000000000518B450 tag: 3 type: 0
 ***            ADC data: block:  2946 ptr: 15747 data: 0xC1D5804B module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32843  ts: 0x157C83743A522C5E OLD: ts: 0x157C83743B109130 dt: 0x0000000000BE64D2 tag: 2 type: 6
 ***            ADC data: block:  2947 ptr:  4715 data: 0xC1C0B930 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 47408  ts: 0x157C83743000001E OLD: ts: 0x157C83743B4E764E dt: 0x000000000B4E7630 tag: 3 type: 0
 ***            ADC data: block:  2959 ptr:  4465 data: 0xC1D5805E module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32862  ts: 0x157C837482ED9B2E OLD: ts: 0x157C837483292CD4 dt: 0x00000000003B91A6 tag: 2 type: 6
 ***            ADC data: block:  2965 ptr:  3385 data: 0xC1C38161 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33121  ts: 0x157C8374E406B2EE OLD: ts: 0x157C8374E856DFBE dt: 0x0000000004502CD0 tag: 3 type: 0
 *** FEE64 discriminator: block:  2966 ptr:  3815 data: 0x84641010 module:  4 information type:  6 information field: 0x00041010 ts: 0x157C83724140840C OLD: ts: 0x157C83724FAD74DE dt: 0x000000000E6CF0D2 tag: 3 type: 0
 ***            ADC data: block:  3007 ptr: 12165 data: 0xC1D481E2 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33250  ts: 0x157C83757266241E OLD: ts: 0x157C837572A80C26 dt: 0x000000000041E808 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  3009 ptr:  7231 data: 0x8748374E module:  7 information type:  4 information field: 0x0008374E ts: 0x157C8374EE000000 OLD: ts: 0x157C83757E000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  3075 ptr: 14985 data: 0xC1D07F61 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32609  ts: 0x157C8376DD47924E OLD: ts: 0x157C8376DD8CADAC dt: 0x0000000000451B5E tag: 2 type: 6
 ***            ADC data: block:  3076 ptr:  8415 data: 0xC1CC8017 module:  7 fail: 0 range: 0 id:   460 channel: 12 adc: 32791  ts: 0x157C8376DA7B92EE OLD: ts: 0x157C8376DF1EF9FE dt: 0x0000000004A36710 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3076 ptr:  8419 data: 0x8748375F module:  7 information type:  4 information field: 0x0008375F ts: 0x157C8375FC000000 OLD: ts: 0x157C8376DC000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***            ADC data: block:  3121 ptr:   395 data: 0xC1F882EF module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33519  ts: 0x157C8377C8FC4B7E OLD: ts: 0x157C8377CC84816E dt: 0x00000000038835F0 tag: 3 type: 0
 ***            ADC data: block:  3157 ptr: 14575 data: 0xC1F88163 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33123  ts: 0x157C8378837B114E OLD: ts: 0x157C837884544838 dt: 0x0000000000D936EA tag: 2 type: 6
 ***            ADC data: block:  3158 ptr:  6709 data: 0xC1D58055 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32853  ts: 0x157C837884F2B06E OLD: ts: 0x157C83788545599E dt: 0x000000000052A930 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3160 ptr:  4783 data: 0x87483775 module:  7 information type:  4 information field: 0x00083775 ts: 0x157C837750000000 OLD: ts: 0x157C837880000000 dt: 0x0000000130000000 tag: 2 type: 5
 ***            ADC data: block:  3161 ptr: 14987 data: 0xC1D27FE6 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32742  ts: 0x157C83775545616E OLD: ts: 0x157C837756CF1A5E dt: 0x000000000189B8F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3163 ptr:  7103 data: 0x874836A3 module:  7 information type:  4 information field: 0x000836A3 ts: 0x157C836A30000000 OLD: ts: 0x157C8378A0000000 dt: 0x0000000E70000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3165 ptr:  5319 data: 0x8748369C module:  7 information type:  4 information field: 0x0008369C ts: 0x157C8369C000120E OLD: ts: 0x157C836A3000120E dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3183 ptr:  6189 data: 0x87483793 module:  7 information type:  4 information field: 0x00083793 ts: 0x157C83793A000000 OLD: ts: 0x157C83795A000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  3183 ptr: 11285 data: 0xC1DE8080 module:  7 fail: 0 range: 0 id:   478 channel: 30 adc: 32896  ts: 0x157C837935CB664E OLD: ts: 0x157C83793B2996EE dt: 0x00000000055E30A0 tag: 3 type: 0
 ***            ADC data: block:  3209 ptr:  9749 data: 0xC1FF81AD module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33197  ts: 0x157C8379AA5E474E OLD: ts: 0x157C8379AACCE960 dt: 0x00000000006EA212 tag: 2 type: 6
 ***            ADC data: block:  3210 ptr:  1941 data: 0xC1EF80C4 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32964  ts: 0x157C8379ABB88CEE OLD: ts: 0x157C8379AC03F09E dt: 0x00000000004B63B0 tag: 3 type: 0
 ***            ADC data: block:  3210 ptr: 16289 data: 0xC1E18151 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33105  ts: 0x157C8379AC042F1E OLD: ts: 0x157C8379AECE874E dt: 0x0000000002CA5830 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3215 ptr: 10881 data: 0x8748378B module:  7 information type:  4 information field: 0x0008378B ts: 0x157C8378B000121E OLD: ts: 0x157C837A0000121E dt: 0x0000000150000000 tag: 2 type: 5
 ***            ADC data: block:  3216 ptr:  7961 data: 0xC1EF81B9 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33209  ts: 0x157C8378B181878E OLD: ts: 0x157C8378B309336E dt: 0x000000000187ABE0 tag: 3 type: 0
 ***            ADC data: block:  3224 ptr:  9363 data: 0xC1E18158 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33112  ts: 0x157C837A21BCB83E OLD: ts: 0x157C837A228EF132 dt: 0x0000000000D238F4 tag: 2 type: 6
 ***            ADC data: block:  3243 ptr:  4961 data: 0xC1D07F09 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32521  ts: 0x157C837A61DBD1DE OLD: ts: 0x157C837A62191A30 dt: 0x00000000003D4852 tag: 2 type: 6
 ***            ADC data: block:  3244 ptr:  7685 data: 0xC1C18156 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33110  ts: 0x157C837A62C57E2E OLD: ts: 0x157C837A6BBCEA0E dt: 0x0000000008F76BE0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3246 ptr: 11299 data: 0x874837A0 module:  7 information type:  4 information field: 0x000837A0 ts: 0x157C837A0C000000 OLD: ts: 0x157C837A8C000000 dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block:  3247 ptr:  8271 data: 0xC1C0816E module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33134  ts: 0x157C837A07D0E42E OLD: ts: 0x157C837A0F0185EE dt: 0x000000000730A1C0 tag: 3 type: 0
 ***            ADC data: block:  3247 ptr:  8273 data: 0xC1E97FE3 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32739  ts: 0x157C837A0309336E OLD: ts: 0x157C837A07D0E42E dt: 0x0000000004C7B0C0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3248 ptr: 15159 data: 0x8748378B module:  7 information type:  4 information field: 0x0008378B ts: 0x157C8378B8000000 OLD: ts: 0x157C837A08000000 dt: 0x0000000150000000 tag: 2 type: 5
 ***            ADC data: block:  3249 ptr:  4229 data: 0xC1C5810D module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33037  ts: 0x157C8378B929702E OLD: ts: 0x157C8378BAC905EE dt: 0x00000000019F95C0 tag: 3 type: 0
 ***            ADC data: block:  3250 ptr: 16231 data: 0xC1FA8152 module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 33106  ts: 0x157C836A1853C0FE OLD: ts: 0x157C836A1C31C14E dt: 0x0000000003DE0050 tag: 3 type: 0
 *** FEE64 discriminator: block:  3251 ptr: 14207 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157C836A144FBF00 OLD: ts: 0x157C836A1B19EDAE dt: 0x0000000006CA2EAE tag: 3 type: 0
 ***            ADC data: block:  3267 ptr:  7395 data: 0xC1F48131 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33073  ts: 0x157C837B12543DEE OLD: ts: 0x157C837B13483F70 dt: 0x0000000000F40182 tag: 2 type: 6
 ***   SYNC100 timestamp: block:  3270 ptr:  9867 data: 0x874837AF module:  7 information type:  4 information field: 0x000837AF ts: 0x157C837AF00008AE OLD: ts: 0x157C837B300008AE dt: 0x0000000040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3272 ptr: 10121 data: 0x874837A9 module:  7 information type:  4 information field: 0x000837A9 ts: 0x157C837A9000329E OLD: ts: 0x157C837AF000329E dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  3304 ptr: 14867 data: 0xC1C58124 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33060  ts: 0x157C837BAD55B30E OLD: ts: 0x157C837BAE791B54 dt: 0x0000000001236846 tag: 2 type: 6
 ***            ADC data: block:  3305 ptr:  6167 data: 0xC1F48159 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33113  ts: 0x157C837BA3E4813E OLD: ts: 0x157C837BAEF30F5E dt: 0x000000000B0E8E20 tag: 3 type: 0
 ***            ADC data: block:  3322 ptr: 15217 data: 0xC1EF820F module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33295  ts: 0x157C837C00CEEE7E OLD: ts: 0x157C837C0190201C dt: 0x0000000000C1319E tag: 2 type: 6
 ***            ADC data: block:  3324 ptr:  8589 data: 0xC1D07EF1 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32497  ts: 0x157C837C02E3016E OLD: ts: 0x157C837C0B534A1E dt: 0x00000000087048B0 tag: 3 type: 0
 ***            ADC data: block:  3337 ptr:  8243 data: 0xC1DB8234 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33332  ts: 0x157C837C78F77D9E OLD: ts: 0x157C837C794E8C24 dt: 0x0000000000570E86 tag: 2 type: 6
 ***            ADC data: block:  3338 ptr:  3453 data: 0xC1D48189 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33161  ts: 0x157C837C753C500E OLD: ts: 0x157C837C7B292AAE dt: 0x0000000005ECDAA0 tag: 3 type: 0
 ***            ADC data: block:  3339 ptr:  1429 data: 0xC1E381CC module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33228  ts: 0x157C837C7741777E OLD: ts: 0x157C837C786322EE dt: 0x000000000121AB70 tag: 3 type: 0
 ***            ADC data: block:  3350 ptr: 14155 data: 0xC1F18278 module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33400  ts: 0x157C837CB1E8075E OLD: ts: 0x157C837CB2BCCBD8 dt: 0x0000000000D4C47A tag: 2 type: 6
 ***            ADC data: block:  3352 ptr:  7475 data: 0xC1D78097 module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32919  ts: 0x157C837CB43E43CE OLD: ts: 0x157C837CBD71907E dt: 0x0000000009334CB0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3356 ptr:  1373 data: 0x874837B4 module:  7 information type:  4 information field: 0x000837B4 ts: 0x157C837B4C000000 OLD: ts: 0x157C837CEC000000 dt: 0x00000001A0000000 tag: 2 type: 5
 ***            ADC data: block:  3356 ptr:  9491 data: 0xC1E180A2 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32930  ts: 0x157C837B415D613E OLD: ts: 0x157C837B4DEE061E dt: 0x000000000C90A4E0 tag: 3 type: 0
 ***            ADC data: block:  3369 ptr:  6089 data: 0xC1EC8217 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33303  ts: 0x157C837D16CA42FE OLD: ts: 0x157C837D16F00EEE dt: 0x000000000025CBF0 tag: 2 type: 6
 *** FEE64 discriminator: block:  3370 ptr:  6655 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157C837D19616D80 OLD: ts: 0x157C837D1CE8B65E dt: 0x00000000038748DE tag: 3 type: 0
 ***            ADC data: block:  3370 ptr: 13483 data: 0xC1EF8180 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33152  ts: 0x157C837D188C173E OLD: ts: 0x157C837D1E7808EE dt: 0x0000000005EBF1B0 tag: 3 type: 0
 ***            ADC data: block:  3385 ptr:  6701 data: 0xC1C38151 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33105  ts: 0x157C837DA62C415E OLD: ts: 0x157C837DA720A5FA dt: 0x0000000000F4649C tag: 2 type: 6
 ***            ADC data: block:  3385 ptr: 11391 data: 0xC1E38211 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33297  ts: 0x157C837DA680BDEE OLD: ts: 0x157C837DA720F9CE dt: 0x0000000000A03BE0 tag: 3 type: 0
 ***            ADC data: block:  3386 ptr:  9367 data: 0xC1E3B9E7 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 47591  ts: 0x157C837DA720F9CE OLD: ts: 0x157C837DA9DDBBFE dt: 0x0000000002BCC230 tag: 3 type: 0
 ***            ADC data: block:  3388 ptr: 13449 data: 0xC1DB8322 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33570  ts: 0x157C837DCAC203AE OLD: ts: 0x157C837DCC4C7A8E dt: 0x00000000018A76E0 tag: 3 type: 0
 ***            ADC data: block:  3389 ptr: 11425 data: 0xC1EF8214 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33300  ts: 0x157C837DCC4CA19E OLD: ts: 0x157C837DCE0138BE dt: 0x0000000001B49720 tag: 3 type: 0
 ***            ADC data: block:  3421 ptr: 13923 data: 0xC1C18164 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33124  ts: 0x157C837E3AC45AFE OLD: ts: 0x157C837E3AFA385E dt: 0x000000000035DD60 tag: 2 type: 6
 ***            ADC data: block:  3422 ptr:  9625 data: 0xC1D78034 module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32820  ts: 0x157C837E32B122EE OLD: ts: 0x157C837E3D1C4CEE dt: 0x000000000A6B2A00 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3423 ptr:  7567 data: 0x874837DD module:  7 information type:  4 information field: 0x000837DD ts: 0x157C837DD6000000 OLD: ts: 0x157C837E36000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3424 ptr: 15329 data: 0x874837CF module:  7 information type:  4 information field: 0x000837CF ts: 0x157C837CF0000000 OLD: ts: 0x157C837DD0000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***            ADC data: block:  3425 ptr: 14003 data: 0xC1D280D6 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32982  ts: 0x157C837E411D556E OLD: ts: 0x157C837E480979F6 dt: 0x0000000006EC2488 tag: 2 type: 3
 ***   SYNC100 timestamp: block:  3427 ptr:  9985 data: 0x874837A9 module:  7 information type:  4 information field: 0x000837A9 ts: 0x157C837A98000000 OLD: ts: 0x157C837E48000000 dt: 0x00000003B0000000 tag: 2 type: 5
 ***            ADC data: block:  3428 ptr:  6935 data: 0xC1F28235 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33333  ts: 0x157C837ADDB9D72E OLD: ts: 0x157C837ADEA77B6E dt: 0x0000000000EDA440 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3439 ptr:  2417 data: 0x874837E7 module:  7 information type:  4 information field: 0x000837E7 ts: 0x157C837E70000A7E OLD: ts: 0x157C837E80000A7E dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  3439 ptr: 13231 data: 0xC1F1816A module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33130  ts: 0x157C837E7108C3AE OLD: ts: 0x157C837E72569CDE dt: 0x00000000014DD930 tag: 3 type: 0
 ***            ADC data: block:  3440 ptr: 11207 data: 0xC1D280A2 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32930  ts: 0x157C837E7325A6FE OLD: ts: 0x157C837E746275FE dt: 0x00000000013CCF00 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3448 ptr:  8023 data: 0x874837E5 module:  7 information type:  4 information field: 0x000837E5 ts: 0x157C837E58000000 OLD: ts: 0x157C837ED8000000 dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block:  3449 ptr:  4473 data: 0xC1F38095 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32917  ts: 0x157C837E58D4600E OLD: ts: 0x157C837E5B12FC3E dt: 0x00000000023E9C30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3462 ptr: 12059 data: 0x874837F1 module:  7 information type:  4 information field: 0x000837F1 ts: 0x157C837F1000101E OLD: ts: 0x157C837F3000101E dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  3469 ptr: 12095 data: 0xC1E9809D module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32925  ts: 0x157C837F50D2E95E OLD: ts: 0x157C837F518F1C6E dt: 0x0000000000BC3310 tag: 2 type: 6
 ***            ADC data: block:  3488 ptr:  8083 data: 0xC1E3818A module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33162  ts: 0x157C837F984AA07E OLD: ts: 0x157C837F986DE8CC dt: 0x000000000023484E tag: 2 type: 6
 ***   SYNC100 timestamp: block:  3489 ptr:  5091 data: 0x874837F4 module:  7 information type:  4 information field: 0x000837F4 ts: 0x157C837F40000EAE OLD: ts: 0x157C837F90000EAE dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block:  3505 ptr:  2799 data: 0xC1C582C1 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33473  ts: 0x157C83803292683E OLD: ts: 0x157C838038B45C9E dt: 0x000000000621F460 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3505 ptr:  9679 data: 0x87483802 module:  7 information type:  4 information field: 0x00083802 ts: 0x157C838024000000 OLD: ts: 0x157C838034000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  3507 ptr:  2985 data: 0xC1F37FA8 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32680  ts: 0x157C838028B483AE OLD: ts: 0x157C83802EE0227E dt: 0x00000000062B9ED0 tag: 3 type: 0
 ***            ADC data: block:  3542 ptr:   115 data: 0xC1D57FF1 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32753  ts: 0x157C8380AD3E21EE OLD: ts: 0x157C8380ADC9A2BE dt: 0x00000000008B80D0 tag: 2 type: 6
 ***            ADC data: block:  3542 ptr: 12897 data: 0xC1D080B1 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32945  ts: 0x157C8380AEE068CE OLD: ts: 0x157C8380AFD73B8E dt: 0x0000000000F6D2C0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3543 ptr:  1971 data: 0x87483804 module:  7 information type:  4 information field: 0x00083804 ts: 0x157C83804000099E OLD: ts: 0x157C8380B000099E dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block:  3543 ptr: 10879 data: 0xC1D3811E module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 33054  ts: 0x157C8380403DDC6E OLD: ts: 0x157C83804214A92E dt: 0x0000000001D6CCC0 tag: 3 type: 0
 ***            ADC data: block:  3602 ptr: 15161 data: 0xC1C88062 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32866  ts: 0x157C8381FA8B732E OLD: ts: 0x157C8381FB5D3BF2 dt: 0x0000000000D1C8C4 tag: 2 type: 6
 ***            ADC data: block:  3603 ptr: 10897 data: 0xC1F78020 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32800  ts: 0x157C8381FB4F425E OLD: ts: 0x157C8381FCFF130E dt: 0x0000000001AFD0B0 tag: 3 type: 0
 ***            ADC data: block:  3604 ptr:  8873 data: 0xC1F8825A module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33370  ts: 0x157C8381F1B81EBE OLD: ts: 0x157C8381FEBAD6CE dt: 0x000000000D02B810 tag: 3 type: 0
 ***            ADC data: block:  3619 ptr:  9355 data: 0xC1D280ED module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 33005  ts: 0x157C838281A1F3EE OLD: ts: 0x157C838282A3994E dt: 0x000000000101A560 tag: 3 type: 0
 *** FEE64 discriminator: block:  3620 ptr:  7331 data: 0x84640010 module:  4 information type:  6 information field: 0x00040010 ts: 0x157C838282FEE146 OLD: ts: 0x157C838284DE3AFE dt: 0x0000000001DF59B8 tag: 3 type: 0
 ***            ADC data: block:  3621 ptr:  9589 data: 0xC1F37FFA module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32762  ts: 0x157C838282A3B88E OLD: ts: 0x157C838288ED6E7E dt: 0x000000000649B5F0 tag: 3 type: 0
 ***            ADC data: block:  3622 ptr: 14369 data: 0xC1F2825E module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33374  ts: 0x157C83828CF450BE OLD: ts: 0x157C83828DFE6210 dt: 0x00000000010A1152 tag: 2 type: 6
 ***            ADC data: block:  3639 ptr:  9251 data: 0xC1E18123 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33059  ts: 0x157C8382C6EF747E OLD: ts: 0x157C8382C76D1D5C dt: 0x00000000007DA8DE tag: 2 type: 6
 ***            ADC data: block:  3641 ptr:   463 data: 0xC1E9800F module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32783  ts: 0x157C8382C8D032CE OLD: ts: 0x157C8382CC3F318E dt: 0x00000000036EFEC0 tag: 3 type: 0
 ***            ADC data: block:  3664 ptr: 15647 data: 0xC1FD82E2 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33506  ts: 0x157C838371411BEE OLD: ts: 0x157C83837AF9A7F0 dt: 0x0000000009B88C02 tag: 2 type: 6
 ***            ADC data: block:  3668 ptr:  2493 data: 0xC1CA8055 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32853  ts: 0x157C83837809F4BE OLD: ts: 0x157C83837E597EDE dt: 0x00000000064F8A20 tag: 3 type: 0
 ***            ADC data: block:  3679 ptr:  4027 data: 0xC1EB80F3 module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33011  ts: 0x157C8383BBADD01E OLD: ts: 0x157C8383BBCCDDEC dt: 0x00000000001F0DCE tag: 2 type: 6
 ***            ADC data: block:  3680 ptr:  3299 data: 0xC1D07F85 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32645  ts: 0x157C8383B7AD9AAE OLD: ts: 0x157C8383BE6E4FAE dt: 0x0000000006C0B500 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3680 ptr:  4577 data: 0x8748383A module:  7 information type:  4 information field: 0x0008383A ts: 0x157C8383A8000000 OLD: ts: 0x157C8383B8000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  3681 ptr:  1275 data: 0xC1E97FDE module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32734  ts: 0x157C8383A4DE3AFE OLD: ts: 0x157C8383AAF7A52E dt: 0x0000000006196A30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3681 ptr: 15083 data: 0x87483826 module:  7 information type:  4 information field: 0x00083826 ts: 0x157C838268000000 OLD: ts: 0x157C8383A8000000 dt: 0x0000000140000000 tag: 2 type: 5
 ***            ADC data: block:  3681 ptr: 15623 data: 0xC1E3820C module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33292  ts: 0x157C8382639479AE OLD: ts: 0x157C8382681A91AE dt: 0x0000000004861800 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3682 ptr: 10081 data: 0x874837F1 module:  7 information type:  4 information field: 0x000837F1 ts: 0x157C837F16000000 OLD: ts: 0x157C838266000000 dt: 0x0000000350000000 tag: 2 type: 5
 ***            ADC data: block:  3684 ptr:  2637 data: 0xC1D681D7 module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 33239  ts: 0x157C8383D6D7777E OLD: ts: 0x157C8383DB992B94 dt: 0x0000000004C1B416 tag: 2 type: 3
 ***            ADC data: block:  3717 ptr:  3577 data: 0xC1C580F4 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33012  ts: 0x157C83845213CCAE OLD: ts: 0x157C83845245D86E dt: 0x0000000000320BC0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3717 ptr: 13123 data: 0x8748383E module:  7 information type:  4 information field: 0x0008383E ts: 0x157C8383E4000000 OLD: ts: 0x157C838454000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block:  3718 ptr: 15903 data: 0xC1D68193 module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 33171  ts: 0x157C8383E245E03E OLD: ts: 0x157C8383E864124E dt: 0x00000000061E3210 tag: 3 type: 0
 ***            ADC data: block:  3721 ptr:  2185 data: 0xC1F88325 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33573  ts: 0x157C83847864395E OLD: ts: 0x157C83847A5B0E6E dt: 0x0000000001F6D510 tag: 2 type: 3
 ***            ADC data: block:  3722 ptr:   163 data: 0xC1E3813F module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33087  ts: 0x157C83847BE1F63E OLD: ts: 0x157C83847F34A72E dt: 0x000000000352B0F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3722 ptr: 10833 data: 0x8748383E module:  7 information type:  4 information field: 0x0008383E ts: 0x157C8383EE000000 OLD: ts: 0x157C83847E000000 dt: 0x0000000090000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  3722 ptr: 14511 data: 0x84641010 module:  4 information type:  6 information field: 0x00041010 ts: 0x157C8383EA6A56D0 OLD: ts: 0x157C8383EED24FCE dt: 0x000000000467F8FE tag: 3 type: 0
 ***            ADC data: block:  3732 ptr: 11025 data: 0xC1F8823F module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33343  ts: 0x157C8384A36794FE OLD: ts: 0x157C8384AA9822CE dt: 0x0000000007308DD0 tag: 3 type: 0
 ***            ADC data: block:  3732 ptr: 11441 data: 0xC1C3814C module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33100  ts: 0x157C8384A378948E OLD: ts: 0x157C8384AB4B6D20 dt: 0x0000000007D2D892 tag: 2 type: 6
 ***            ADC data: block:  3733 ptr:  9001 data: 0xC1F1824C module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33356  ts: 0x157C8384A299EA6E OLD: ts: 0x157C8384A6D442AE dt: 0x00000000043A5840 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3733 ptr: 14447 data: 0x87483848 module:  7 information type:  4 information field: 0x00083848 ts: 0x157C838484000000 OLD: ts: 0x157C8384A4000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  3749 ptr: 14385 data: 0xC1DC8006 module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32774  ts: 0x157C83855A0B3C6E OLD: ts: 0x157C83855B3F115A dt: 0x000000000133D4EC tag: 2 type: 6
 ***            ADC data: block:  3750 ptr:  7493 data: 0xC1C081A5 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33189  ts: 0x157C83855257A3BE OLD: ts: 0x157C83855C0946BE dt: 0x0000000009B1A300 tag: 3 type: 0
 ***            ADC data: block:  3752 ptr:  3447 data: 0xC1C080C2 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 32962  ts: 0x157C838555DF8FEE OLD: ts: 0x157C838558469F1E dt: 0x0000000002670F30 tag: 3 type: 0
 ***            ADC data: block:  3753 ptr: 15773 data: 0xC1E180C0 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32960  ts: 0x157C83855C0946BE OLD: ts: 0x157C83855CE87EFE dt: 0x0000000000DF3840 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3755 ptr:  9769 data: 0x874837E6 module:  7 information type:  4 information field: 0x000837E6 ts: 0x157C837E6000567E OLD: ts: 0x157C83856000567E dt: 0x0000000700000000 tag: 2 type: 5
 ***            ADC data: block:  3757 ptr: 12919 data: 0xC1F282C3 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33475  ts: 0x157C837E67C1E2FE OLD: ts: 0x157C837E687DF3CE dt: 0x0000000000BC10D0 tag: 3 type: 0
 ***            ADC data: block:  3757 ptr: 12921 data: 0xC1E3812A module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33066  ts: 0x157C837E61F9464E OLD: ts: 0x157C837E67C1E2FE dt: 0x0000000005C89CB0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3759 ptr:  6011 data: 0x874837AE module:  7 information type:  4 information field: 0x000837AE ts: 0x157C837AE8000000 OLD: ts: 0x157C837E68000000 dt: 0x0000000380000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3759 ptr: 12985 data: 0x8748361F module:  7 information type:  4 information field: 0x0008361F ts: 0x157C8361FC000000 OLD: ts: 0x157C837AEC000000 dt: 0x00000018F0000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  3760 ptr:  6851 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157C8361F293CCFC OLD: ts: 0x157C8361FE06F3AE dt: 0x000000000B7326B2 tag: 3 type: 0
 ***            ADC data: block:  3784 ptr: 10885 data: 0xC1D5801C module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32796  ts: 0x157C8385D742A7EE OLD: ts: 0x157C8385D7E154E8 dt: 0x00000000009EACFA tag: 2 type: 6
 ***            ADC data: block:  3785 ptr:   515 data: 0xC1D58019 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32793  ts: 0x157C8385D380CCDE OLD: ts: 0x157C8385D8710DFE dt: 0x0000000004F04120 tag: 3 type: 0
 ***            ADC data: block:  3785 ptr:  8857 data: 0xC1E081F4 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33268  ts: 0x157C8385D5545C2E OLD: ts: 0x157C8385D91A1A7A dt: 0x0000000003C5BE4C tag: 2 type: 6
 ***            ADC data: block:  3799 ptr:   219 data: 0xC1CD8145 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33093  ts: 0x157C83868430CC2E OLD: ts: 0x157C83868564706E dt: 0x000000000133A440 tag: 2 type: 6
 ***            ADC data: block:  3849 ptr:   721 data: 0xC1DF80D3 module:  7 fail: 0 range: 0 id:   479 channel: 31 adc: 32979  ts: 0x157C838727E9A3AE OLD: ts: 0x157C838727E9B04C dt: 0x0000000000000C9E tag: 3 type: 0
 ***            ADC data: block:  3849 ptr:  9295 data: 0xC1C08199 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33177  ts: 0x157C838724DE963E OLD: ts: 0x157C838729B5341E dt: 0x0000000004D69DE0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  3850 ptr:  6743 data: 0x87483867 module:  7 information type:  4 information field: 0x00083867 ts: 0x157C838678000000 OLD: ts: 0x157C838728000000 dt: 0x00000000B0000000 tag: 2 type: 5
 ***            ADC data: block:  3858 ptr: 10139 data: 0xC1E18054 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32852  ts: 0x157C8387AB1A99AE OLD: ts: 0x157C8387ABDD09B2 dt: 0x0000000000C27004 tag: 2 type: 6
 ***            ADC data: block:  3906 ptr: 15569 data: 0xC1FD8128 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33064  ts: 0x157C83886255438E OLD: ts: 0x157C83886307CE2E dt: 0x0000000000B28AA0 tag: 3 type: 0
 *** FEE64 discriminator: block:  3907 ptr: 13545 data: 0x88640010 module:  8 information type:  6 information field: 0x00040010 ts: 0x157C83886325B15A OLD: ts: 0x157C8388658405AE dt: 0x00000000025E5454 tag: 3 type: 0
 ***            ADC data: block:  3914 ptr:  8781 data: 0xC1C180B0 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 32944  ts: 0x157C8388E307E59E OLD: ts: 0x157C8388E976341E dt: 0x00000000066E4E80 tag: 3 type: 0
 ***            ADC data: block:  3977 ptr: 16319 data: 0xC1C181E9 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33257  ts: 0x157C8389F2849FBE OLD: ts: 0x157C8389F62D7E6E dt: 0x0000000003A8DEB0 tag: 3 type: 0
 ***            ADC data: block:  3986 ptr: 13381 data: 0xC1CB8179 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33145  ts: 0x157C838A1153048E OLD: ts: 0x157C838A1275484A dt: 0x00000000012243BC tag: 2 type: 6
 ***            ADC data: block:  3989 ptr:  8257 data: 0xC1CD8183 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33155  ts: 0x157C838A26687BAE OLD: ts: 0x157C838A2F8FA78E dt: 0x0000000009272BE0 tag: 3 type: 0
 *** FEE64 discriminator: block:  3990 ptr:  6233 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157C838A2258126A OLD: ts: 0x157C838A29A6D5FE dt: 0x00000000074EC394 tag: 3 type: 0
 ***            ADC data: block:  3991 ptr: 10183 data: 0xC1DB820D module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33293  ts: 0x157C838A24001D7E OLD: ts: 0x157C838A2AC8235E dt: 0x0000000006C805E0 tag: 3 type: 0
 ***            ADC data: block:  3999 ptr: 13727 data: 0xC1DC80FB module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 33019  ts: 0x157C838A44913E2E OLD: ts: 0x157C838A47E3456E dt: 0x0000000003520740 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4002 ptr: 11187 data: 0x874838A2 module:  7 information type:  4 information field: 0x000838A2 ts: 0x157C838A2E000000 OLD: ts: 0x157C838A6E000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  4002 ptr: 11335 data: 0xC1C780D8 module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32984  ts: 0x157C838A29A6FD0E OLD: ts: 0x157C838A2E0E2E6E dt: 0x0000000004673160 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4003 ptr:  6701 data: 0x87483859 module:  7 information type:  4 information field: 0x00083859 ts: 0x157C83859C000000 OLD: ts: 0x157C838A2C000000 dt: 0x0000000490000000 tag: 2 type: 5
 ***            ADC data: block:  4003 ptr:  9311 data: 0xC1DB81E1 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33249  ts: 0x157C838591BA7FBE OLD: ts: 0x157C83859C89162E dt: 0x000000000ACE9670 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4004 ptr:  2741 data: 0x87483857 module:  7 information type:  4 information field: 0x00083857 ts: 0x157C838574000000 OLD: ts: 0x157C838594000000 dt: 0x0000000020000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  4005 ptr:  5267 data: 0x84641010 module:  4 information type:  6 information field: 0x00041010 ts: 0x157C8385731ADAF8 OLD: ts: 0x157C83857990116E dt: 0x0000000006753676 tag: 3 type: 0
 ***            ADC data: block:  4017 ptr:  5437 data: 0xC1F380FE module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 33022  ts: 0x157C838A9D04CFBE OLD: ts: 0x157C838A9D3D31AE dt: 0x00000000003861F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4017 ptr:  9631 data: 0x874838A7 module:  7 information type:  4 information field: 0x000838A7 ts: 0x157C838A7E000000 OLD: ts: 0x157C838A9E000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  4019 ptr: 15743 data: 0xC1E68122 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 33058  ts: 0x157C838A8D3D397E OLD: ts: 0x157C838A8FF630CE dt: 0x0000000002B8F750 tag: 3 type: 0
 ***            ADC data: block:  4044 ptr: 10295 data: 0xC1E38233 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33331  ts: 0x157C838B3B9CF4CE OLD: ts: 0x157C838B3C000000 dt: 0x0000000000630B32 tag: 2 type: 4
 ***            ADC data: block:  4045 ptr:  8205 data: 0xC1C87FF3 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32755  ts: 0x157C838B33F6DAAE OLD: ts: 0x157C838B3E8AFC9E dt: 0x000000000A9421F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4046 ptr:   373 data: 0x874838AF module:  7 information type:  4 information field: 0x000838AF ts: 0x157C838AF6000000 OLD: ts: 0x157C838B36000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  4048 ptr:  2395 data: 0x874838A7 module:  7 information type:  4 information field: 0x000838A7 ts: 0x157C838A7000334E OLD: ts: 0x157C838AF000334E dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block:  4052 ptr:   797 data: 0xC1F281C2 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33218  ts: 0x157C838B6AD12F4E OLD: ts: 0x157C838B6B0B741A dt: 0x00000000003A44CC tag: 2 type: 6
 ***   SYNC100 timestamp: block:  4059 ptr: 15623 data: 0x874838B6 module:  7 information type:  4 information field: 0x000838B6 ts: 0x157C838B6004C7CE OLD: ts: 0x157C838BF004C7CE dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  4062 ptr: 13159 data: 0xC1F77FC6 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32710  ts: 0x157C838B630F8E5E OLD: ts: 0x157C838B6A8132BE dt: 0x000000000771A460 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4063 ptr:   413 data: 0x874838A7 module:  7 information type:  4 information field: 0x000838A7 ts: 0x157C838A74000000 OLD: ts: 0x157C838B64000000 dt: 0x00000000F0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  4065 ptr:  1539 data: 0x87483857 module:  7 information type:  4 information field: 0x00083857 ts: 0x157C83857E000000 OLD: ts: 0x157C838A7E000000 dt: 0x0000000500000000 tag: 2 type: 5
 *** FEE64 discriminator: block:  4065 ptr:  7091 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157C838571728084 OLD: ts: 0x157C83857F4B00AE dt: 0x000000000DD8802A tag: 3 type: 0
 ***            ADC data: block:  4109 ptr: 14499 data: 0xC1F881A1 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33185  ts: 0x157C838C99ECB46E OLD: ts: 0x157C838C9A000000 dt: 0x0000000000134B92 tag: 2 type: 4
 ***            ADC data: block:  4110 ptr: 12839 data: 0xC1C68194 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33172  ts: 0x157C838C95B959FE OLD: ts: 0x157C838C9CDC621E dt: 0x0000000007230820 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4111 ptr:  7013 data: 0x874838C1 module:  7 information type:  4 information field: 0x000838C1 ts: 0x157C838C18000000 OLD: ts: 0x157C838C98000000 dt: 0x0000000080000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  4112 ptr:  1495 data: 0x874838C0 module:  7 information type:  4 information field: 0x000838C0 ts: 0x157C838C0000127E OLD: ts: 0x157C838C1000127E dt: 0x0000000010000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  4171 ptr:  5699 data: 0x874838D3 module:  7 information type:  4 information field: 0x000838D3 ts: 0x157C838D3A000000 OLD: ts: 0x157C838DEA000000 dt: 0x00000000B0000000 tag: 2 type: 5
 ***            ADC data: block:  4171 ptr: 12601 data: 0xC1C28200 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33280  ts: 0x157C838D301BE0EE OLD: ts: 0x157C838D3B5D228E dt: 0x000000000B4141A0 tag: 3 type: 0
 ***            ADC data: block:  4234 ptr:  6919 data: 0xC1C87FAA module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32682  ts: 0x157C838F32A6236E OLD: ts: 0x157C838F32CBB4D0 dt: 0x0000000000259162 tag: 2 type: 6
 ***            ADC data: block:  4236 ptr:  3561 data: 0xC1F7819F module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33183  ts: 0x157C838E817AC3CE OLD: ts: 0x157C838E85D0615E dt: 0x0000000004559D90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4236 ptr: 13715 data: 0x874838C0 module:  7 information type:  4 information field: 0x000838C0 ts: 0x157C838C04000000 OLD: ts: 0x157C838E84000000 dt: 0x0000000280000000 tag: 2 type: 5
 ***            ADC data: block:  4251 ptr: 14621 data: 0xC1D07FDA module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32730  ts: 0x157C838FABB2D8CE OLD: ts: 0x157C838FABC190EE dt: 0x00000000000EB820 tag: 2 type: 6
 ***            ADC data: block:  4252 ptr: 13877 data: 0xC1FD8230 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33328  ts: 0x157C838FADD7047E OLD: ts: 0x157C838FAED07C1E dt: 0x0000000000F977A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4253 ptr:  7373 data: 0x874838F9 module:  7 information type:  4 information field: 0x000838F9 ts: 0x157C838F9000BA7E OLD: ts: 0x157C838FA000BA7E dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  4253 ptr: 11857 data: 0xC1E68086 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32902  ts: 0x157C838F9093EFDE OLD: ts: 0x157C838F910B641E dt: 0x0000000000777440 tag: 3 type: 0
 ***            ADC data: block:  4278 ptr:  4103 data: 0xC1C680C5 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 32965  ts: 0x157C83902277828E OLD: ts: 0x157C83902381D206 dt: 0x00000000010A4F78 tag: 2 type: 6
 ***            ADC data: block:  4280 ptr:   299 data: 0xC1FA812F module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 33071  ts: 0x157C83902537D33E OLD: ts: 0x157C83902B4659EE dt: 0x00000000060E86B0 tag: 3 type: 0
 ***            ADC data: block:  4300 ptr:  1341 data: 0xC1F78231 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33329  ts: 0x157C8390B5BEAA5E OLD: ts: 0x157C8390B622F054 dt: 0x00000000006445F6 tag: 2 type: 6
 ***            ADC data: block:  4333 ptr: 12129 data: 0xC1C082B3 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33459  ts: 0x157C839146ABEC3E OLD: ts: 0x157C8391478ADF5C dt: 0x0000000000DEF31E tag: 2 type: 6
 ***            ADC data: block:  4334 ptr:  3879 data: 0xC1CB8134 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33076  ts: 0x157C8391437B728E OLD: ts: 0x157C8391486FEB2E dt: 0x0000000004F478A0 tag: 3 type: 0
 ***            ADC data: block:  4335 ptr: 16203 data: 0xC1F4807C module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32892  ts: 0x157C839143D7D03E OLD: ts: 0x157C83914EC7CFEE dt: 0x000000000AEFFFB0 tag: 3 type: 0
 ***            ADC data: block:  4354 ptr:  7175 data: 0xC1F7804D module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32845  ts: 0x157C8391CD9FFE5E OLD: ts: 0x157C8391CEE025DE dt: 0x0000000001402780 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4355 ptr:   763 data: 0x8748391A module:  7 information type:  4 information field: 0x0008391A ts: 0x157C8391A000451E OLD: ts: 0x157C8391C000451E dt: 0x0000000020000000 tag: 2 type: 5
... 2318 more lines ...
Attachment 2: R2_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 ***            ADC data: block:     1 ptr:  5477 data: 0xC1E39CDA module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 40154  ts: 0x157CC966A64D2ABC OLD: ts: 0x157CC966AC9A969C dt: 0x00000000064D6BE0 tag: 3 type: 0
 ***            ADC data: block:    10 ptr:  7957 data: 0xC1FC80A3 module:  7 fail: 0 range: 0 id:   508 channel: 60 adc: 32931  ts: 0x157CC97ABC9A9E6C OLD: ts: 0x157CC97ABDE0AD9C dt: 0x0000000001460F30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    11 ptr:  5407 data: 0x874C966B module:  7 information type:  4 information field: 0x000C966B ts: 0x157CC966B00008BC OLD: ts: 0x157CC97AB00008BC dt: 0x0000001400000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    12 ptr: 12395 data: 0x874C960C module:  7 information type:  4 information field: 0x000C960C ts: 0x157CC960C6000000 OLD: ts: 0x157CC966B6000000 dt: 0x00000005F0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    24 ptr:  8551 data: 0x874C960D module:  7 information type:  4 information field: 0x000C960D ts: 0x157CC960D0000000 OLD: ts: 0x157CC97B30000000 dt: 0x0000001A60000000 tag: 2 type: 5
 ***            ADC data: block:    34 ptr: 14519 data: 0xC1F4813C module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33084  ts: 0x157CC97BA4A99ECC OLD: ts: 0x157CC97BAD06D89C dt: 0x00000000085D39D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    36 ptr:  4661 data: 0x874C960D module:  7 information type:  4 information field: 0x000C960D ts: 0x157CC960DA000000 OLD: ts: 0x157CC97BAA000000 dt: 0x0000001AD0000000 tag: 2 type: 5
 ***            ADC data: block:    47 ptr: 10505 data: 0xC1EC81A1 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33185  ts: 0x157CC97C21EEF58C OLD: ts: 0x157CC97C25D13D6C dt: 0x0000000003E247E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    48 ptr:  3189 data: 0x874C960E module:  7 information type:  4 information field: 0x000C960E ts: 0x157CC960E4000000 OLD: ts: 0x157CC97C24000000 dt: 0x0000001B40000000 tag: 2 type: 5
 ***            ADC data: block:    51 ptr:  4209 data: 0xC0439E5C module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 40540  ts: 0x157CC960E5D13E02 OLD: ts: 0x157CC960EF0DA1DC dt: 0x00000000093C63DA tag: 3 type: 0
 ***   SYNC100 timestamp: block:    60 ptr: 10519 data: 0x874C960F module:  7 information type:  4 information field: 0x000C960F ts: 0x157CC960F000105C OLD: ts: 0x157CC97C9000105C dt: 0x0000001BA0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    73 ptr: 14199 data: 0x874C9610 module:  7 information type:  4 information field: 0x000C9610 ts: 0x157CC961000029EC OLD: ts: 0x157CC97D100029EC dt: 0x0000001C10000000 tag: 2 type: 5
 ***            ADC data: block:    75 ptr: 11245 data: 0xC1D6A3DB module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 41947  ts: 0x157CC96104F56C9C OLD: ts: 0x157CC96106CDFC2C dt: 0x0000000001D88F90 tag: 3 type: 0
 ***            ADC data: block:    84 ptr: 13025 data: 0xC1F281C1 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33217  ts: 0x157CC97D86CDFC2C OLD: ts: 0x157CC97D8C89AAFC dt: 0x0000000005BBAED0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    86 ptr:  2253 data: 0x874C9610 module:  7 information type:  4 information field: 0x000C9610 ts: 0x157CC9610C000000 OLD: ts: 0x157CC97D8C000000 dt: 0x0000001C80000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    97 ptr: 13277 data: 0x874C9611 module:  7 information type:  4 information field: 0x000C9611 ts: 0x157CC96116000000 OLD: ts: 0x157CC97E06000000 dt: 0x0000001CF0000000 tag: 2 type: 5
 ***            ADC data: block:   110 ptr:  1203 data: 0xC1E97F8D module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32653  ts: 0x157CC97E7138489C OLD: ts: 0x157CC97E7A7A827C dt: 0x00000000094239E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   112 ptr:  5101 data: 0x874C9612 module:  7 information type:  4 information field: 0x000C9612 ts: 0x157CC9612A000000 OLD: ts: 0x157CC97E7A000000 dt: 0x0000001D50000000 tag: 2 type: 5
 ***            ADC data: block:   113 ptr: 11279 data: 0xC04EA04D module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 41037  ts: 0x157CC9612A7A8312 OLD: ts: 0x157CC9612F47020C dt: 0x0000000004CC7EFA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   123 ptr:  1653 data: 0x874C9613 module:  7 information type:  4 information field: 0x000C9613 ts: 0x157CC9613000016C OLD: ts: 0x157CC97EF000016C dt: 0x0000001DC0000000 tag: 2 type: 5
 ***            ADC data: block:   126 ptr:  7379 data: 0xC1E59FBB module:  7 fail: 0 range: 0 id:   485 channel: 37 adc: 40891  ts: 0x157CC961347DEC1C OLD: ts: 0x157CC9613CBC88BC dt: 0x00000000083E9CA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   135 ptr: 15431 data: 0x874C9613 module:  7 information type:  4 information field: 0x000C9613 ts: 0x157CC9613E000000 OLD: ts: 0x157CC97F6E000000 dt: 0x0000001E30000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   147 ptr: 12937 data: 0x874C9614 module:  7 information type:  4 information field: 0x000C9614 ts: 0x157CC96148000000 OLD: ts: 0x157CC97FE8000000 dt: 0x0000001EA0000000 tag: 2 type: 5
 ***            ADC data: block:   159 ptr: 16335 data: 0xC1C58102 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33026  ts: 0x157CC9805416AEFC OLD: ts: 0x157CC9805A0192CC dt: 0x0000000005EAE3D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   162 ptr:  2123 data: 0x874C9615 module:  7 information type:  4 information field: 0x000C9615 ts: 0x157CC9615C000000 OLD: ts: 0x157CC9805C000000 dt: 0x0000001F00000000 tag: 2 type: 5
 ***            ADC data: block:   172 ptr: 12809 data: 0xC1E18197 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33175  ts: 0x157CC980D1BFC0FC OLD: ts: 0x157CC980D2CDEB9C dt: 0x00000000010E2AA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   173 ptr: 15185 data: 0x874C9616 module:  7 information type:  4 information field: 0x000C9616 ts: 0x157CC96166000000 OLD: ts: 0x157CC980D6000000 dt: 0x0000001F70000000 tag: 2 type: 5
 ***            ADC data: block:   176 ptr:  4719 data: 0xC044A540 module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 42304  ts: 0x157CC96162CDEC32 OLD: ts: 0x157CC9616F2E90EC dt: 0x000000000C60A4BA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   185 ptr: 10317 data: 0x874C9617 module:  7 information type:  4 information field: 0x000C9617 ts: 0x157CC96170000000 OLD: ts: 0x157CC98140000000 dt: 0x0000001FD0000000 tag: 2 type: 5
 ***            ADC data: block:   188 ptr: 14841 data: 0xC04EA060 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 41056  ts: 0x157CC9617928FEB2 OLD: ts: 0x157CC9617C65DDEC dt: 0x00000000033CDF3A tag: 3 type: 0
 ***   SYNC100 timestamp: block:   199 ptr:  1305 data: 0x874C9618 module:  7 information type:  4 information field: 0x000C9618 ts: 0x157CC96180001BAC OLD: ts: 0x157CC981C0001BAC dt: 0x0000002040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   200 ptr:  2029 data: 0x874C95AE module:  7 information type:  4 information field: 0x000C95AE ts: 0x157CC95AE0000000 OLD: ts: 0x157CC96180000000 dt: 0x00000006A0000000 tag: 2 type: 5
 ***            ADC data: block:   200 ptr: 11715 data: 0xC1E59F81 module:  7 fail: 0 range: 0 id:   485 channel: 37 adc: 40833  ts: 0x157CC95AE1F629DC OLD: ts: 0x157CC95AE249229C dt: 0x000000000052F8C0 tag: 3 type: 0
 ***            ADC data: block:   209 ptr: 13579 data: 0xC1D58057 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32855  ts: 0x157CC98232492A6C OLD: ts: 0x157CC982398985AC dt: 0x0000000007405B40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   211 ptr: 15195 data: 0x874C95AE module:  7 information type:  4 information field: 0x000C95AE ts: 0x157CC95AEA000000 OLD: ts: 0x157CC9823A000000 dt: 0x0000002750000000 tag: 2 type: 5
 ***            ADC data: block:   213 ptr:  5489 data: 0xC1E2A092 module:  7 fail: 0 range: 0 id:   482 channel: 34 adc: 41106  ts: 0x157CC95AE98985AC OLD: ts: 0x157CC95AEF54568C dt: 0x0000000005CAD0E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   222 ptr: 11565 data: 0x874C95AF module:  7 information type:  4 information field: 0x000C95AF ts: 0x157CC95AF000A3EC OLD: ts: 0x157CC982B000A3EC dt: 0x00000027C0000000 tag: 2 type: 5
 ***            ADC data: block:   225 ptr:  2213 data: 0xC1E7A081 module:  7 fail: 0 range: 0 id:   487 channel: 39 adc: 41089  ts: 0x157CC95AF11CF11C OLD: ts: 0x157CC95AF8E659AC dt: 0x0000000007C96890 tag: 3 type: 0
 ***            ADC data: block:   234 ptr:  5573 data: 0xC1D2812A module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 33066  ts: 0x157CC98328E678EC OLD: ts: 0x157CC98329E9F5CC dt: 0x0000000001037CE0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   235 ptr: 12551 data: 0x874C95AF module:  7 information type:  4 information field: 0x000C95AF ts: 0x157CC95AFE000000 OLD: ts: 0x157CC9832E000000 dt: 0x0000002830000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   247 ptr: 10957 data: 0x874C95B0 module:  7 information type:  4 information field: 0x000C95B0 ts: 0x157CC95B08000000 OLD: ts: 0x157CC983A8000000 dt: 0x00000028A0000000 tag: 2 type: 5
 ***            ADC data: block:   250 ptr: 10065 data: 0xC045A02C module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 41004  ts: 0x157CC95B117BF2A2 OLD: ts: 0x157CC95B1363823C dt: 0x0000000001E78F9A tag: 3 type: 0
 ***            ADC data: block:   259 ptr: 11767 data: 0xC1C582BD module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33469  ts: 0x157CC9841363A17C OLD: ts: 0x157CC98417D82D6C dt: 0x0000000004748BF0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   262 ptr:  2993 data: 0x874C95B1 module:  7 information type:  4 information field: 0x000C95B1 ts: 0x157CC95B1C000000 OLD: ts: 0x157CC9841C000000 dt: 0x0000002900000000 tag: 2 type: 5
 ***            ADC data: block:   263 ptr:  3677 data: 0xC1FFA313 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 41747  ts: 0x157CC95B17D82D6C OLD: ts: 0x157CC95B1FF16FDC dt: 0x0000000008194270 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   272 ptr:  6217 data: 0x874C95B2 module:  7 information type:  4 information field: 0x000C95B2 ts: 0x157CC95B20005C2C OLD: ts: 0x157CC98490005C2C dt: 0x0000002970000000 tag: 2 type: 5
 ***            ADC data: block:   275 ptr:    55 data: 0xC0419F6F module:  1 fail: 0 range: 0 id:    65 channel:  1 adc: 40815  ts: 0x157CC95B20A3B3E2 OLD: ts: 0x157CC95B2A094C9C dt: 0x00000000096598BA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   285 ptr: 12223 data: 0x874C95B3 module:  7 information type:  4 information field: 0x000C95B3 ts: 0x157CC95B30000000 OLD: ts: 0x157CC98500000000 dt: 0x00000029D0000000 tag: 2 type: 5
 ***            ADC data: block:   295 ptr: 15473 data: 0xC1C7819B module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 33179  ts: 0x157CC9857412AA6C OLD: ts: 0x157CC9857FCCF35C dt: 0x000000000BBA48F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   297 ptr:  9095 data: 0x874C95B3 module:  7 information type:  4 information field: 0x000C95B3 ts: 0x157CC95B3A000000 OLD: ts: 0x157CC9857A000000 dt: 0x0000002A40000000 tag: 2 type: 5
 ***            ADC data: block:   308 ptr: 11479 data: 0xC1D07F9E module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32670  ts: 0x157CC985F183FAFC OLD: ts: 0x157CC985F7602FEC dt: 0x0000000005DC34F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   309 ptr:  5215 data: 0x874C95B4 module:  7 information type:  4 information field: 0x000C95B4 ts: 0x157CC95B44000000 OLD: ts: 0x157CC985F4000000 dt: 0x0000002AB0000000 tag: 2 type: 5
 ***            ADC data: block:   312 ptr:  3389 data: 0xC045A028 module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 41000  ts: 0x157CC95B47603082 OLD: ts: 0x157CC95B4EAC24FC dt: 0x00000000074BF47A tag: 3 type: 0
 ***   SYNC100 timestamp: block:   321 ptr: 13307 data: 0x874C95B5 module:  7 information type:  4 information field: 0x000C95B5 ts: 0x157CC95B50005ABC OLD: ts: 0x157CC98670005ABC dt: 0x0000002B20000000 tag: 2 type: 5
 ***            ADC data: block:   324 ptr:  2051 data: 0xC1F09FD9 module:  7 fail: 0 range: 0 id:   496 channel: 48 adc: 40921  ts: 0x157CC95B5164E97C OLD: ts: 0x157CC95B591ED3BC dt: 0x0000000007B9EA40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   335 ptr:   493 data: 0x874C95B6 module:  7 information type:  4 information field: 0x000C95B6 ts: 0x157CC95B60000EBC OLD: ts: 0x157CC986E0000EBC dt: 0x0000002B80000000 tag: 2 type: 5
 *** FEE64 discriminator: block:   344 ptr: 14695 data: 0x87610020 module:  7 information type:  6 information field: 0x00010020 ts: 0x157CC9875292C7DC OLD: ts: 0x157CC9875E1A3B8C dt: 0x000000000B8773B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   347 ptr:  7031 data: 0x874C95B6 module:  7 information type:  4 information field: 0x000C95B6 ts: 0x157CC95B6C000000 OLD: ts: 0x157CC9875C000000 dt: 0x0000002BF0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   357 ptr:  9833 data: 0x874C95B7 module:  7 information type:  4 information field: 0x000C95B7 ts: 0x157CC95B7000284C OLD: ts: 0x157CC987D000284C dt: 0x0000002C60000000 tag: 2 type: 5
 ***            ADC data: block:   360 ptr:  2279 data: 0xC1D09D8B module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 40331  ts: 0x157CC95B75AD29FC OLD: ts: 0x157CC95B7962B6FC dt: 0x0000000003B58D00 tag: 3 type: 0
 ***            ADC data: block:   369 ptr:  5883 data: 0xC1C4800E module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32782  ts: 0x157CC9884962EDAC OLD: ts: 0x157CC9884D41BE4C dt: 0x0000000003DED0A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   371 ptr:  1713 data: 0x874C95B8 module:  7 information type:  4 information field: 0x000C95B8 ts: 0x157CC95B80000000 OLD: ts: 0x157CC98840000000 dt: 0x0000002CC0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   382 ptr: 13055 data: 0x874C95B8 module:  7 information type:  4 information field: 0x000C95B8 ts: 0x157CC95B8A000000 OLD: ts: 0x157CC988CA000000 dt: 0x0000002D40000000 tag: 2 type: 5
 ***            ADC data: block:   949 ptr:     7 data: 0xC0AF5DD7 module:  2 fail: 0 range: 0 id:   175 channel: 47 adc: 24023  ts: 0x157CC9A593F069E0 OLD: ts: 0x157CC9A59CB63300 dt: 0x0000000008C5C920 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   949 ptr:   781 data: 0x824C9A52 module:  2 information type:  4 information field: 0x000C9A52 ts: 0x157CC9A52C000000 OLD: ts: 0x157CC9A59C000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3022 ptr: 10885 data: 0x814CA0E0 module:  1 information type:  4 information field: 0x000CA0E0 ts: 0x157CCA0E00CE5EC2 OLD: ts: 0x157CCA0F80CE5EC2 dt: 0x0000000180000000 tag: 2 type: 5
 ***            ADC data: block:  8941 ptr:     7 data: 0xC0935FCF module:  2 fail: 0 range: 0 id:   147 channel: 19 adc: 24527  ts: 0x157CCB3DC32A17D0 OLD: ts: 0x157CCB3DCBE16200 dt: 0x0000000008B74A30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8941 ptr:   367 data: 0x824CB3D5 module:  2 information type:  4 information field: 0x000CB3D5 ts: 0x157CCB3D56000000 OLD: ts: 0x157CCB3DC6000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 11956 ptr:   211 data: 0x874CBD75 module:  7 information type:  4 information field: 0x000CBD75 ts: 0x157CCBD75041DF2C OLD: ts: 0x157CCBD7C041DF2C dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 13889 ptr:     7 data: 0xC1FFA34E module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 41806  ts: 0x157CCC3A6419FA8C OLD: ts: 0x157CCC3A6E09016C dt: 0x0000000009EF06E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 13889 ptr:  1019 data: 0x874CC39F module:  7 information type:  4 information field: 0x000CC39F ts: 0x157CCC39FE000000 OLD: ts: 0x157CCC3A6E000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 17602 ptr:     7 data: 0xC20F6111 module:  8 fail: 0 range: 0 id:   527 channel: 15 adc: 24849  ts: 0x157CCCF801D6514A OLD: ts: 0x157CCCF80A8EA51A dt: 0x0000000008B853D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 17602 ptr:   785 data: 0x884CCF79 module:  8 information type:  4 information field: 0x000CCF79 ts: 0x157CCCF79A000000 OLD: ts: 0x157CCCF80A000000 dt: 0x0000000070000000 tag: 2 type: 5
 *** ERROR: READ I/O error:       5002
                   blocks:      18050
          ADC data format:  146892420 (   36529.6 Hz)
        Other data format:     864931 (     215.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:     422478 (     105.1 Hz)
                              WR48-63:     422478 (     105.1 Hz)
                           FEE64 disc:      19975 (       5.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          5 (       0.0 Hz)

                Timewarps:        ADC:         33 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:         39 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     4021.191 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                6.879
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000             4838.139
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12180938      70390          0          0          0          0      35195      35195          0          0          0          0
  1   12191107      70376          0          0          0          0      35188      35188          0          0          0          0
  2   12207827      70458          0          0          0          0      35229      35229          0          0          0          0
  3   12189283      70384          0          0          0          0      35192      35192          0          0          0          0
  4   12183260      70382          0          0          0          0      35191      35191          0          0          0          0
  5   12183902      70390          0          0          0          0      35195      35195          0          0          0          0
  6   12193309      70394          0          0          0          0      35197      35197          0          0          0          0
  7   12809402      90549          0          0          0          0      35287      35287      19975          0          0          5
  8   12191658      70432          0          0          0          0      35216      35216          0          0          0          0
  9   12184261      70394          0          0          0          0      35197      35197          0          0          0          0
 10   12189839      70390          0          0          0          0      35195      35195          0          0          0          0
 11   12187634      70392          0          0          0          0      35196      35196          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          7          0          0          1          0          0          0          0          0
  2          2          0          0          2          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7         23          0          0         35          0          1          0          0          0
  8          1          0          0          1          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 3463.661s (    5.211 blocks/s,   0.326 Mb/s)
Attachment 3: R3_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 ***   SYNC100 timestamp: block:     3 ptr:  1017 data: 0x874CD5F1 module:  7 information type:  4 information field: 0x000CD5F1 ts: 0x157CCD5F100B650C OLD: ts: 0x157CCD61E00B650C dt: 0x00000002D0000000 tag: 2 type: 5
 ***            ADC data: block:     3 ptr: 11625 data: 0xC20F6F01 module:  8 fail: 0 range: 0 id:   527 channel: 15 adc: 28417  ts: 0x157CCD5F109FE1FA OLD: ts: 0x157CCD5F150E896C dt: 0x00000000046EA772 tag: 3 type: 0
 ***            ADC data: block:     7 ptr:  1647 data: 0xC2206DD7 module:  8 fail: 0 range: 0 id:   544 channel: 32 adc: 28119  ts: 0x157CCD61E533A80A OLD: ts: 0x157CCD61EA1C22CA dt: 0x0000000004E87AC0 tag: 3 type: 0
 ***            ADC data: block:     7 ptr:  7083 data: 0xC2086EC4 module:  8 fail: 0 range: 0 id:   520 channel:  8 adc: 28356  ts: 0x157CCD61E7B570EA OLD: ts: 0x157CCD61EA1C275C dt: 0x000000000266B672 tag: 3 type: 0
 ***            ADC data: block:     7 ptr: 15995 data: 0xC1D2907A module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 36986  ts: 0x157CCD61EA1C275C OLD: ts: 0x157CCD61EBE2903A dt: 0x0000000001C668DE tag: 3 type: 0
 ***            ADC data: block:     9 ptr:  7469 data: 0xC1CE90A0 module:  7 fail: 0 range: 0 id:   462 channel: 14 adc: 37024  ts: 0x157CCD61E09FE68C OLD: ts: 0x157CCD61EEE261FC dt: 0x000000000E427B70 tag: 3 type: 0
 ***            ADC data: block:    13 ptr:  4735 data: 0xC064934C module:  1 fail: 0 range: 0 id:   100 channel: 36 adc: 37708  ts: 0x157CCD61F5C5D462 OLD: ts: 0x157CCD61F6FFA7D2 dt: 0x000000000139D370 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    14 ptr:  1769 data: 0x814CD616 module:  1 information type:  4 information field: 0x000CD616 ts: 0x157CCD616C000000 OLD: ts: 0x157CCD61FC000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:    14 ptr:  2711 data: 0xC06C9233 module:  1 fail: 0 range: 0 id:   108 channel: 44 adc: 37427  ts: 0x157CCD6166FFA7D2 OLD: ts: 0x157CCD616C748DB2 dt: 0x000000000574E5E0 tag: 3 type: 0
 ***            ADC data: block:    16 ptr:  6357 data: 0xC0EE9267 module:  3 fail: 0 range: 0 id:   238 channel: 46 adc: 37479  ts: 0x157CCD61F7300A40 OLD: ts: 0x157CCD61FCE7ABE0 dt: 0x0000000005B7A1A0 tag: 3 type: 0
 ***            ADC data: block:    17 ptr:  4333 data: 0xC1C88F73 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 36723  ts: 0x157CCD61FCE7AD5C OLD: ts: 0x157CCD61FDEDDEC0 dt: 0x0000000001063164 tag: 3 type: 0
 ***            ADC data: block:    55 ptr:  8123 data: 0xC2BD62A6 module: 10 fail: 0 range: 0 id:   701 channel: 61 adc: 25254  ts: 0x157CCD6245C1C2A6 OLD: ts: 0x157CCD6245C1C33C dt: 0x0000000000000096 tag: 3 type: 0
 ***            ADC data: block:    70 ptr:  9537 data: 0xC0679354 module:  1 fail: 0 range: 0 id:   103 channel: 39 adc: 37716  ts: 0x157CCD627A151EC2 OLD: ts: 0x157CCD627A15216A dt: 0x00000000000002A8 tag: 3 type: 0
 ***            ADC data: block:    74 ptr:  8325 data: 0xC05B91CB module:  1 fail: 0 range: 0 id:    91 channel: 27 adc: 37323  ts: 0x157CCD62800CA5A2 OLD: ts: 0x157CCD628BB3A4B2 dt: 0x000000000BA6FF10 tag: 3 type: 0
 ***            ADC data: block:    79 ptr:  5583 data: 0xC2936E63 module: 10 fail: 0 range: 0 id:   659 channel: 19 adc: 28259  ts: 0x157CCD62AB2566D6 OLD: ts: 0x157CCD62AD83AF62 dt: 0x00000000025E488C tag: 3 type: 0
 ***   SYNC100 timestamp: block:    79 ptr: 14459 data: 0x884CD628 module:  8 information type:  4 information field: 0x000CD628 ts: 0x157CCD628E000000 OLD: ts: 0x157CCD62AE000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:    80 ptr: 11453 data: 0xC20A6E03 module:  8 fail: 0 range: 0 id:   522 channel: 10 adc: 28163  ts: 0x157CCD629075015A OLD: ts: 0x157CCD629E0CDD46 dt: 0x000000000D97DBEC tag: 3 type: 0
 ***   SYNC100 timestamp: block:    80 ptr: 15649 data: 0x8A4CD5DB module: 10 information type:  4 information field: 0x000CD5DB ts: 0x157CCD5DB004CD16 OLD: ts: 0x157CCD629004CD16 dt: 0x00000004E0000000 tag: 2 type: 5
 ***            ADC data: block:    81 ptr:  6725 data: 0xC2BC60DC module: 10 fail: 0 range: 0 id:   700 channel: 60 adc: 24796  ts: 0x157CCD5DB074FD86 OLD: ts: 0x157CCD5DB37AB9F6 dt: 0x000000000305BC70 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    94 ptr:  3267 data: 0x844CD5E7 module:  4 information type:  4 information field: 0x000CD5E7 ts: 0x157CCD5E7005AA22 OLD: ts: 0x157CCD62F005AA22 dt: 0x0000000480000000 tag: 2 type: 5
 ***            ADC data: block:    94 ptr: 15569 data: 0xC0439145 module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 37189  ts: 0x157CCD5E753B1F02 OLD: ts: 0x157CCD5E76D34D82 dt: 0x0000000001982E80 tag: 3 type: 0
 ***            ADC data: block:    95 ptr:  2943 data: 0xC1236EDD module:  4 fail: 0 range: 0 id:   291 channel: 35 adc: 28381  ts: 0x157CCD5E75D3EB42 OLD: ts: 0x157CCD5E75D3EC32 dt: 0x00000000000000F0 tag: 3 type: 0
 ***            ADC data: block:   113 ptr:  4537 data: 0xC18E6CFD module:  6 fail: 0 range: 0 id:   398 channel: 14 adc: 27901  ts: 0x157CCD634ED49A6E OLD: ts: 0x157CCD634F16B5FE dt: 0x0000000000421B90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   113 ptr:  6987 data: 0x864CD5E7 module:  6 information type:  4 information field: 0x000CD5E7 ts: 0x157CCD5E7005A82E OLD: ts: 0x157CCD634005A82E dt: 0x00000004D0000000 tag: 2 type: 5
 ***            ADC data: block:   114 ptr: 15627 data: 0xC12C6D56 module:  4 fail: 0 range: 0 id:   300 channel: 44 adc: 27990  ts: 0x157CCD63522E6222 OLD: ts: 0x157CCD63553BADAE dt: 0x00000000030D4B8C tag: 3 type: 0
 ***            ADC data: block:   115 ptr: 15813 data: 0xC1A76C00 module:  6 fail: 0 range: 0 id:   423 channel: 39 adc: 27648  ts: 0x157CCD63699ACC7E OLD: ts: 0x157CCD636ADD9942 dt: 0x000000000142CCC4 tag: 2 type: 3
 ***   SYNC100 timestamp: block:   116 ptr: 13207 data: 0x864CD562 module:  6 information type:  4 information field: 0x000CD562 ts: 0x157CCD56200C2EAE OLD: ts: 0x157CCD63600C2EAE dt: 0x0000000D40000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   117 ptr:  7475 data: 0x864CD51E module:  6 information type:  4 information field: 0x000CD51E ts: 0x157CCD51EA000000 OLD: ts: 0x157CCD562A000000 dt: 0x0000000440000000 tag: 2 type: 5
 ***            ADC data: block:   117 ptr: 11769 data: 0xC1B16CEE module:  6 fail: 0 range: 0 id:   433 channel: 49 adc: 27886  ts: 0x157CCD51E7964A4E OLD: ts: 0x157CCD51EC56972E dt: 0x0000000004C04CE0 tag: 3 type: 0
 ***            ADC data: block:   118 ptr:  7701 data: 0xC18C6F26 module:  6 fail: 0 range: 0 id:   396 channel: 12 adc: 28454  ts: 0x157CCD51E21CA73E OLD: ts: 0x157CCD51EDF8432E dt: 0x000000000BDB9BF0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   118 ptr:  7803 data: 0x864CD425 module:  6 information type:  4 information field: 0x000CD425 ts: 0x157CCD425E000000 OLD: ts: 0x157CCD51EE000000 dt: 0x0000000F90000000 tag: 2 type: 5
 ***            ADC data: block:   141 ptr: 14999 data: 0xC1956DFB module:  6 fail: 0 range: 0 id:   405 channel: 21 adc: 28155  ts: 0x157CCD63B085567E OLD: ts: 0x157CCD63B0855872 dt: 0x00000000000001F4 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   147 ptr:  9033 data: 0x874CD61F module:  7 information type:  4 information field: 0x000CD61F ts: 0x157CCD61F0043D1C OLD: ts: 0x157CCD63B0043D1C dt: 0x00000001C0000000 tag: 2 type: 5
 ***            ADC data: block:   157 ptr:  3129 data: 0xC0106E53 module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 28243  ts: 0x157CCD63D0BD57FC OLD: ts: 0x157CCD63DDC6C83C dt: 0x000000000D097040 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   166 ptr: 12243 data: 0x894CD5FD module:  9 information type:  4 information field: 0x000CD5FD ts: 0x157CCD5FD00C5366 OLD: ts: 0x157CCD63E00C5366 dt: 0x0000000410000000 tag: 2 type: 5
 ***            ADC data: block:   179 ptr:  3049 data: 0xC0206D14 module:  0 fail: 0 range: 0 id:    32 channel: 32 adc: 27924  ts: 0x157CCD63FC0E64DC OLD: ts: 0x157CCD63FC0E6982 dt: 0x00000000000004A6 tag: 3 type: 0
 ***            ADC data: block:   188 ptr: 12583 data: 0xC1E791FE module:  7 fail: 0 range: 0 id:   487 channel: 39 adc: 37374  ts: 0x157CCD6410FFD54C OLD: ts: 0x157CCD6410FFD736 dt: 0x00000000000001EA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   196 ptr: 16375 data: 0x894CD640 module:  9 information type:  4 information field: 0x000CD640 ts: 0x157CCD6400000000 OLD: ts: 0x157CCD6420000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:   217 ptr:  4009 data: 0xC1036E6A module:  4 fail: 0 range: 0 id:   259 channel:  3 adc: 28266  ts: 0x157CCD6451700462 OLD: ts: 0x157CCD6459916BE0 dt: 0x000000000821677E tag: 3 type: 0
 ***            ADC data: block:   233 ptr:  4177 data: 0xC14D908B module:  5 fail: 0 range: 0 id:   333 channel: 13 adc: 37003  ts: 0x157CCD646B91CE2C OLD: ts: 0x157CCD646FCF04BC dt: 0x00000000043D3690 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   233 ptr: 13675 data: 0x854CD63B module:  5 information type:  4 information field: 0x000CD63B ts: 0x157CCD63B0000000 OLD: ts: 0x157CCD6460000000 dt: 0x00000000B0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   236 ptr:   523 data: 0x824CD646 module:  2 information type:  4 information field: 0x000CD646 ts: 0x157CCD64600E1C20 OLD: ts: 0x157CCD64700E1C20 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   236 ptr: 12243 data: 0xC09B6C9E module:  2 fail: 0 range: 0 id:   155 channel: 27 adc: 27806  ts: 0x157CCD6464764AC0 OLD: ts: 0x157CCD64657C5690 dt: 0x0000000001060BD0 tag: 3 type: 0
 ***            ADC data: block:   241 ptr:  2179 data: 0xC14C8F5B module:  5 fail: 0 range: 0 id:   332 channel: 12 adc: 36699  ts: 0x157CCD647067B2AC OLD: ts: 0x157CCD647D3B5466 dt: 0x000000000CD3A1BA tag: 3 type: 0
 ***            ADC data: block:   280 ptr:  7731 data: 0xC2C09061 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 36961  ts: 0x157CCD64F4061738 OLD: ts: 0x157CCD64FAE33DE8 dt: 0x0000000006DD26B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   300 ptr: 11361 data: 0x834CD620 module:  3 information type:  4 information field: 0x000CD620 ts: 0x157CCD620C000000 OLD: ts: 0x157CCD652C000000 dt: 0x0000000320000000 tag: 2 type: 5
 ***            ADC data: block:   301 ptr:  5673 data: 0xC1489055 module:  5 fail: 0 range: 0 id:   328 channel:  8 adc: 36949  ts: 0x157CCD62106A90FC OLD: ts: 0x157CCD6210F3F710 dt: 0x0000000000896614 tag: 3 type: 0
 ***            ADC data: block:   309 ptr:  7677 data: 0xC0DA9274 module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 37492  ts: 0x157CCD65206A95C0 OLD: ts: 0x157CCD652E703710 dt: 0x000000000E05A150 tag: 3 type: 0
 ***            ADC data: block:   368 ptr: 14253 data: 0xC0AE6B66 module:  2 fail: 0 range: 0 id:   174 channel: 46 adc: 27494  ts: 0x157CCD65A1D3D4D0 OLD: ts: 0x157CCD65AC82D810 dt: 0x000000000AAF0340 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   368 ptr: 14523 data: 0x824CD648 module:  2 information type:  4 information field: 0x000CD648 ts: 0x157CCD6482000000 OLD: ts: 0x157CCD65A2000000 dt: 0x0000000120000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   390 ptr: 11451 data: 0x834CD653 module:  3 information type:  4 information field: 0x000CD653 ts: 0x157CCD65300C2C00 OLD: ts: 0x157CCD65F00C2C00 dt: 0x00000000C0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   391 ptr:  9417 data: 0x834CD425 module:  3 information type:  4 information field: 0x000CD425 ts: 0x157CCD425001DD20 OLD: ts: 0x157CCD653001DD20 dt: 0x00000022E0000000 tag: 2 type: 5
 ***            ADC data: block:   392 ptr:  4083 data: 0xC0806F2F module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 28463  ts: 0x157CCD4254F181F0 OLD: ts: 0x157CCD42552395A0 dt: 0x00000000003213B0 tag: 3 type: 0
 ***            ADC data: block:   405 ptr:  3703 data: 0xC0906D92 module:  2 fail: 0 range: 0 id:   144 channel: 16 adc: 28050  ts: 0x157CCD660BDAE500 OLD: ts: 0x157CCD660CB0B522 dt: 0x0000000000D5D022 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   408 ptr:  9195 data: 0x824CD65C module:  2 information type:  4 information field: 0x000CD65C ts: 0x157CCD65C0052320 OLD: ts: 0x157CCD6610052320 dt: 0x0000000050000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   409 ptr:  5777 data: 0x824CD616 module:  2 information type:  4 information field: 0x000CD616 ts: 0x157CCD61600B54C0 OLD: ts: 0x157CCD65C00B54C0 dt: 0x0000000460000000 tag: 2 type: 5
 ***            ADC data: block:   409 ptr: 14383 data: 0xC07992D5 module:  1 fail: 0 range: 0 id:   121 channel: 57 adc: 37589  ts: 0x157CCD6611B0BF12 OLD: ts: 0x157CCD6613179FAC dt: 0x000000000166E09A tag: 2 type: 3
 ***   SYNC100 timestamp: block:   410 ptr:  7279 data: 0x814CD62B module:  1 information type:  4 information field: 0x000CD62B ts: 0x157CCD62B6000000 OLD: ts: 0x157CCD6616000000 dt: 0x0000000360000000 tag: 2 type: 5
 ***            ADC data: block:   410 ptr: 12359 data: 0xC2F4AD06 module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 44294  ts: 0x157CCD62B3225078 OLD: ts: 0x157CCD62B85F7862 dt: 0x00000000053D27EA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   424 ptr: 11245 data: 0x834CD661 module:  3 information type:  4 information field: 0x000CD661 ts: 0x157CCD66100813A0 OLD: ts: 0x157CCD66300813A0 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:   427 ptr:  2895 data: 0xC2A36F0A module: 10 fail: 0 range: 0 id:   675 channel: 35 adc: 28426  ts: 0x157CCD663599A196 OLD: ts: 0x157CCD663A18A286 dt: 0x00000000047F00F0 tag: 3 type: 0
 ***            ADC data: block:   428 ptr:   871 data: 0xC1C49484 module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 38020  ts: 0x157CCD663A18A31C OLD: ts: 0x157CCD663C4862B6 dt: 0x00000000022FBF9A tag: 3 type: 0
 ***            ADC data: block:   430 ptr:  1249 data: 0xC06390B2 module:  1 fail: 0 range: 0 id:    99 channel: 35 adc: 37042  ts: 0x157CCD663926B7E2 OLD: ts: 0x157CCD663E275DC2 dt: 0x000000000500A5E0 tag: 3 type: 0
 ***            ADC data: block:   430 ptr: 15597 data: 0xC0409047 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 36935  ts: 0x157CCD66340C6732 OLD: ts: 0x157CCD663FD580D2 dt: 0x000000000BC919A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   432 ptr:  4231 data: 0x814CD5BF module:  1 information type:  4 information field: 0x000CD5BF ts: 0x157CCD5BFE000000 OLD: ts: 0x157CCD663E000000 dt: 0x0000000A40000000 tag: 2 type: 5
 ***            ADC data: block:   432 ptr:  4787 data: 0xC07798BE module:  1 fail: 0 range: 0 id:   119 channel: 55 adc: 39102  ts: 0x157CCD5BFE275DC2 OLD: ts: 0x157CCD5BFE407E82 dt: 0x00000000001920C0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   434 ptr:  5293 data: 0x834CD663 module:  3 information type:  4 information field: 0x000CD663 ts: 0x157CCD663A000000 OLD: ts: 0x157CCD664A000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   434 ptr:  6857 data: 0xC2807136 module: 10 fail: 0 range: 0 id:   640 channel:  0 adc: 28982  ts: 0x157CCD66316F6B46 OLD: ts: 0x157CCD663AC099A0 dt: 0x0000000009512E5A tag: 3 type: 0
 ***            ADC data: block:   452 ptr: 11591 data: 0xC2046AB8 module:  8 fail: 0 range: 0 id:   516 channel:  4 adc: 27320  ts: 0x157CCD66700BF77A OLD: ts: 0x157CCD667FE7BE5A dt: 0x000000000FDBC6E0 tag: 3 type: 0
 ***            ADC data: block:   485 ptr:  9671 data: 0xC2326612 module:  8 fail: 0 range: 0 id:   562 channel: 50 adc: 26130  ts: 0x157CCD66D299E8DA OLD: ts: 0x157CCD66D299ECD6 dt: 0x00000000000003FC tag: 3 type: 0
 ***   SYNC100 timestamp: block:   514 ptr:  9769 data: 0x814CD66C module:  1 information type:  4 information field: 0x000CD66C ts: 0x157CCD66C00B2AB2 OLD: ts: 0x157CCD67100B2AB2 dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block:   539 ptr: 12675 data: 0xC22A6E6F module:  8 fail: 0 range: 0 id:   554 channel: 42 adc: 28271  ts: 0x157CCD674DB0F4FA OLD: ts: 0x157CCD674DB0F73E dt: 0x0000000000000244 tag: 3 type: 0
 ***            ADC data: block:   553 ptr: 16109 data: 0xC1176F2D module:  4 fail: 0 range: 0 id:   279 channel: 23 adc: 28461  ts: 0x157CCD676FA3FBA2 OLD: ts: 0x157CCD676FA3FC92 dt: 0x00000000000000F0 tag: 3 type: 0
 ***            ADC data: block:   568 ptr:  9165 data: 0xC00D6F6B module:  0 fail: 0 range: 0 id:    13 channel: 13 adc: 28523  ts: 0x157CCD678E48F6EC OLD: ts: 0x157CCD678F066B4C dt: 0x0000000000BD7460 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   568 ptr: 12719 data: 0x804CD63E module:  0 information type:  4 information field: 0x000CD63E ts: 0x157CCD63E0038F9C OLD: ts: 0x157CCD6780038F9C dt: 0x00000003A0000000 tag: 2 type: 5
 ***            ADC data: block:   576 ptr: 14153 data: 0xC1FB92C3 module:  7 fail: 0 range: 0 id:   507 channel: 59 adc: 37571  ts: 0x157CCD679A6A8EEC OLD: ts: 0x157CCD679D863B0C dt: 0x00000000031BAC20 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   577 ptr:  5377 data: 0x874CD63C module:  7 information type:  4 information field: 0x000CD63C ts: 0x157CCD63CE000000 OLD: ts: 0x157CCD679E000000 dt: 0x00000003D0000000 tag: 2 type: 5
 ***            ADC data: block:   580 ptr:  5945 data: 0xC07A9286 module:  1 fail: 0 range: 0 id:   122 channel: 58 adc: 37510  ts: 0x157CCD67A45398B2 OLD: ts: 0x157CCD67A4539AEC dt: 0x000000000000023A tag: 3 type: 0
 ***   SYNC100 timestamp: block:   586 ptr: 11319 data: 0x804CD67A module:  0 information type:  4 information field: 0x000CD67A ts: 0x157CCD67A00789BC OLD: ts: 0x157CCD67B00789BC dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   587 ptr:  3511 data: 0xC18E6CE7 module:  6 fail: 0 range: 0 id:   398 channel: 14 adc: 27879  ts: 0x157CCD67A2965F5E OLD: ts: 0x157CCD67A4075F4C dt: 0x000000000170FFEE tag: 3 type: 0
 ***            ADC data: block:   608 ptr: 13227 data: 0xC12F6D8A module:  4 fail: 0 range: 0 id:   303 channel: 47 adc: 28042  ts: 0x157CCD67F4355892 OLD: ts: 0x157CCD67F7802A32 dt: 0x00000000034AD1A0 tag: 3 type: 0
 ***            ADC data: block:   653 ptr:  4581 data: 0xC1916BE6 module:  6 fail: 0 range: 0 id:   401 channel: 17 adc: 27622  ts: 0x157CCD68500B939E OLD: ts: 0x157CCD68500B9592 dt: 0x00000000000001F4 tag: 3 type: 0
 ***            ADC data: block:   657 ptr:   617 data: 0xC03A6CB5 module:  0 fail: 0 range: 0 id:    58 channel: 58 adc: 27829  ts: 0x157CCD685030D84C OLD: ts: 0x157CCD6857AEF30C dt: 0x00000000077E1AC0 tag: 3 type: 0
 ***            ADC data: block:   688 ptr: 11357 data: 0xC0996B76 module:  2 fail: 0 range: 0 id:   153 channel: 25 adc: 27510  ts: 0x157CCD68B910A7A0 OLD: ts: 0x157CCD68BD97D4E0 dt: 0x0000000004872D40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   689 ptr:  1313 data: 0x824CD684 module:  2 information type:  4 information field: 0x000CD684 ts: 0x157CCD684C000000 OLD: ts: 0x157CCD68BC000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block:   689 ptr:  9333 data: 0xC1096DCE module:  4 fail: 0 range: 0 id:   265 channel:  9 adc: 28110  ts: 0x157CCD684D97D5B2 OLD: ts: 0x157CCD684FBF7090 dt: 0x0000000002279ADE tag: 3 type: 0
 ***            ADC data: block:   724 ptr:  7401 data: 0xC0916C94 module:  2 fail: 0 range: 0 id:   145 channel: 17 adc: 27796  ts: 0x157CCD68FAFD1610 OLD: ts: 0x157CCD68FDF8FB7C dt: 0x0000000002FBE56C tag: 3 type: 0
 ***   SYNC100 timestamp: block:   726 ptr: 14211 data: 0x824CD68C module:  2 information type:  4 information field: 0x000CD68C ts: 0x157CCD68CE000000 OLD: ts: 0x157CCD690E000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   727 ptr:  7757 data: 0x824CD685 module:  2 information type:  4 information field: 0x000CD685 ts: 0x157CCD68500B8CF0 OLD: ts: 0x157CCD68D00B8CF0 dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block:   728 ptr:  5179 data: 0xC1056D38 module:  4 fail: 0 range: 0 id:   261 channel:  5 adc: 27960  ts: 0x157CCD68516EDEA2 OLD: ts: 0x157CCD6857164120 dt: 0x0000000005A7627E tag: 3 type: 0
 ***   SYNC100 timestamp: block:   733 ptr: 12951 data: 0x854CD64D module:  5 information type:  4 information field: 0x000CD64D ts: 0x157CCD64D00BEB9C OLD: ts: 0x157CCD69100BEB9C dt: 0x0000000440000000 tag: 2 type: 5
 ***            ADC data: block:   734 ptr:  3491 data: 0xC10B6AB6 module:  4 fail: 0 range: 0 id:   267 channel: 11 adc: 27318  ts: 0x157CCD64D0D3D172 OLD: ts: 0x157CCD64D335EAFC dt: 0x000000000262198A tag: 3 type: 0
 ***            ADC data: block:   736 ptr: 10661 data: 0xC15B90BB module:  5 fail: 0 range: 0 id:   347 channel: 27 adc: 37051  ts: 0x157CCD6910D3D35C OLD: ts: 0x157CCD691E3174DC dt: 0x000000000D5DA180 tag: 3 type: 0
 ***            ADC data: block:   743 ptr: 11259 data: 0xC26B93A4 module:  9 fail: 0 range: 0 id:   619 channel: 43 adc: 37796  ts: 0x157CCD694043B766 OLD: ts: 0x157CCD69460BC4A8 dt: 0x0000000005C80D42 tag: 2 type: 3
 ***   SYNC100 timestamp: block:   743 ptr: 15009 data: 0x894CD643 module:  9 information type:  4 information field: 0x000CD643 ts: 0x157CCD6432000000 OLD: ts: 0x157CCD6942000000 dt: 0x0000000510000000 tag: 2 type: 5
 ***            ADC data: block:   758 ptr:  3015 data: 0xC0D591B1 module:  3 fail: 0 range: 0 id:   213 channel: 21 adc: 37297  ts: 0x157CCD69716F5AC0 OLD: ts: 0x157CCD6977093FE0 dt: 0x000000000599E520 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   758 ptr:  8479 data: 0x834CD664 module:  3 information type:  4 information field: 0x000CD664 ts: 0x157CCD6644000000 OLD: ts: 0x157CCD6974000000 dt: 0x0000000330000000 tag: 2 type: 5
 ***            ADC data: block:   759 ptr:   991 data: 0xC0EE926A module:  3 fail: 0 range: 0 id:   238 channel: 46 adc: 37482  ts: 0x157CCD6647093FE0 OLD: ts: 0x157CCD66481E1410 dt: 0x000000000114D430 tag: 3 type: 0
 ***            ADC data: block:   763 ptr:  6967 data: 0xC2C09066 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 36966  ts: 0x157CCD6982953E38 OLD: ts: 0x157CCD69869BC748 dt: 0x0000000004068910 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   763 ptr: 10041 data: 0x8B4CD650 module: 11 information type:  4 information field: 0x000CD650 ts: 0x157CCD6504000000 OLD: ts: 0x157CCD6984000000 dt: 0x0000000480000000 tag: 2 type: 5
 ***            ADC data: block:   764 ptr:  4943 data: 0xC2489039 module:  9 fail: 0 range: 0 id:   584 channel:  8 adc: 36921  ts: 0x157CCD65069BCDF6 OLD: ts: 0x157CCD6509EC0EC8 dt: 0x00000000035040D2 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   767 ptr: 10805 data: 0x8B4CD698 module: 11 information type:  4 information field: 0x000CD698 ts: 0x157CCD698C000000 OLD: ts: 0x157CCD699C000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   767 ptr: 13577 data: 0xC2469386 module:  9 fail: 0 range: 0 id:   582 channel:  6 adc: 37766  ts: 0x157CCD6984270F26 OLD: ts: 0x157CCD698D4A9038 dt: 0x0000000009238112 tag: 3 type: 0
 ***            ADC data: block:   770 ptr:   683 data: 0xC170927B module:  5 fail: 0 range: 0 id:   368 channel: 48 adc: 37499  ts: 0x157CCD699B9DD3CC OLD: ts: 0x157CCD699CCE254C dt: 0x0000000001305180 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   773 ptr:   573 data: 0x8B4CD699 module: 11 information type:  4 information field: 0x000CD699 ts: 0x157CCD6996000000 OLD: ts: 0x157CCD69A6000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   773 ptr: 11769 data: 0xC1498FBB module:  5 fail: 0 range: 0 id:   329 channel:  9 adc: 36795  ts: 0x157CCD6992F34BDC OLD: ts: 0x157CCD699B501248 dt: 0x00000000085CC66C tag: 3 type: 0
 ***            ADC data: block:   793 ptr:  9841 data: 0xC1439035 module:  5 fail: 0 range: 0 id:   323 channel:  3 adc: 36917  ts: 0x157CCD69D1915C9C OLD: ts: 0x157CCD69DA9E8C2C dt: 0x00000000090D2F90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   793 ptr: 15067 data: 0x854CD69B module:  5 information type:  4 information field: 0x000CD69B ts: 0x157CCD69B4000000 OLD: ts: 0x157CCD69D4000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:   794 ptr: 11507 data: 0xC15F91BC module:  5 fail: 0 range: 0 id:   351 channel: 31 adc: 37308  ts: 0x157CCD69BB099A6C OLD: ts: 0x157CCD69BB467608 dt: 0x00000000003CDB9C tag: 3 type: 0
 ***            ADC data: block:   827 ptr:  5501 data: 0xC2539085 module:  9 fail: 0 range: 0 id:   595 channel: 19 adc: 36997  ts: 0x157CCD6A38E41036 OLD: ts: 0x157CCD6A38E41158 dt: 0x0000000000000122 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   834 ptr:  3847 data: 0x824CD6A0 module:  2 information type:  4 information field: 0x000CD6A0 ts: 0x157CCD6A000751B0 OLD: ts: 0x157CCD6A400751B0 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:   840 ptr: 12203 data: 0xC09C6D9C module:  2 fail: 0 range: 0 id:   156 channel: 28 adc: 28060  ts: 0x157CCD6A574572A0 OLD: ts: 0x157CCD6A5B60B1F0 dt: 0x00000000041B3F50 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   841 ptr:  1623 data: 0x824CD6A4 module:  2 information type:  4 information field: 0x000CD6A4 ts: 0x157CCD6A4A000000 OLD: ts: 0x157CCD6A5A000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:   841 ptr: 10179 data: 0xC046936B module:  1 fail: 0 range: 0 id:    70 channel:  6 adc: 37739  ts: 0x157CCD6A4B60B3B2 OLD: ts: 0x157CCD6A4DF42BF0 dt: 0x000000000293783E tag: 3 type: 0
 ***            ADC data: block:   859 ptr:  4101 data: 0xC29B6EBB module: 10 fail: 0 range: 0 id:   667 channel: 27 adc: 28347  ts: 0x157CCD6A8933AF56 OLD: ts: 0x157CCD6A8933B2F8 dt: 0x00000000000003A2 tag: 3 type: 0
 ***            ADC data: block:   863 ptr:  9515 data: 0xC0EE9266 module:  3 fail: 0 range: 0 id:   238 channel: 46 adc: 37478  ts: 0x157CCD6A919D7D70 OLD: ts: 0x157CCD6A919D7DC0 dt: 0x0000000000000050 tag: 3 type: 0
 ***            ADC data: block:   866 ptr:  4201 data: 0xC2DD922A module: 11 fail: 0 range: 0 id:   733 channel: 29 adc: 37418  ts: 0x157CCD6A9050E708 OLD: ts: 0x157CCD6A983CD1F8 dt: 0x0000000007EBEAF0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   866 ptr: 12033 data: 0x8B4CD69B module: 11 information type:  4 information field: 0x000CD69B ts: 0x157CCD69B4000000 OLD: ts: 0x157CCD6A94000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   874 ptr:  5319 data: 0x824CD6A5 module:  2 information type:  4 information field: 0x000CD6A5 ts: 0x157CCD6A500A61C0 OLD: ts: 0x157CCD6AB00A61C0 dt: 0x0000000060000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   876 ptr: 11857 data: 0x824CD46A module:  2 information type:  4 information field: 0x000CD46A ts: 0x157CCD46AA000000 OLD: ts: 0x157CCD6A6A000000 dt: 0x00000023C0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   878 ptr:  8631 data: 0x824CD427 module:  2 information type:  4 information field: 0x000CD427 ts: 0x157CCD42700C8070 OLD: ts: 0x157CCD46B00C8070 dt: 0x0000000440000000 tag: 2 type: 5
 ***            ADC data: block:   880 ptr:  5057 data: 0xC2846B78 module: 10 fail: 0 range: 0 id:   644 channel:  4 adc: 27512  ts: 0x157CCD427A4E5D16 OLD: ts: 0x157CCD427D984670 dt: 0x000000000349E95A tag: 3 type: 0
 ***            ADC data: block:   881 ptr:  3891 data: 0xC0896DD3 module:  2 fail: 0 range: 0 id:   137 channel:  9 adc: 28115  ts: 0x157CCD4284B1EEC0 OLD: ts: 0x157CCD428A4E5D16 dt: 0x00000000059C6E56 tag: 3 type: 0
 ***            ADC data: block:   883 ptr: 14775 data: 0xC2A26D18 module: 10 fail: 0 range: 0 id:   674 channel: 34 adc: 27928  ts: 0x157CCD6AD60013B6 OLD: ts: 0x157CCD6AD823C4CA dt: 0x000000000223B114 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   884 ptr: 11973 data: 0x884CD66B module:  8 information type:  4 information field: 0x000CD66B ts: 0x157CCD66B00DB3AA OLD: ts: 0x157CCD6AD00DB3AA dt: 0x0000000420000000 tag: 2 type: 5
 ***            ADC data: block:   884 ptr: 12853 data: 0xC2BD65EC module: 10 fail: 0 range: 0 id:   701 channel: 61 adc: 26092  ts: 0x157CCD66B16FD0D6 OLD: ts: 0x157CCD66B6007176 dt: 0x000000000490A0A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   885 ptr:  1879 data: 0x8A4CD664 module: 10 information type:  4 information field: 0x000CD664 ts: 0x157CCD6644000000 OLD: ts: 0x157CCD66B4000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   886 ptr:  1675 data: 0x8A4CD513 module: 10 information type:  4 information field: 0x000CD513 ts: 0x157CCD5130087376 OLD: ts: 0x157CCD6640087376 dt: 0x0000001510000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   886 ptr: 15075 data: 0x8A4CD427 module: 10 information type:  4 information field: 0x000CD427 ts: 0x157CCD42700C8106 OLD: ts: 0x157CCD51300C8106 dt: 0x0000000EC0000000 tag: 2 type: 5
 ***            ADC data: block:   892 ptr:  1097 data: 0xC0E4934C module:  3 fail: 0 range: 0 id:   228 channel: 36 adc: 37708  ts: 0x157CCD6AF4CB9F10 OLD: ts: 0x157CCD6AFCA7E510 dt: 0x0000000007DC4600 tag: 3 type: 0
 ***            ADC data: block:   896 ptr: 14647 data: 0xC0F79847 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 38983  ts: 0x157CCD6B211B1210 OLD: ts: 0x157CCD6B2669C936 dt: 0x00000000054EB726 tag: 2 type: 3
 ***   SYNC100 timestamp: block:   896 ptr: 16199 data: 0x834CD661 module:  3 information type:  4 information field: 0x000CD661 ts: 0x157CCD6612000000 OLD: ts: 0x157CCD6B22000000 dt: 0x0000000510000000 tag: 2 type: 5
 ***            ADC data: block:   897 ptr: 12623 data: 0xC0D39181 module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 37249  ts: 0x157CCD6615239D70 OLD: ts: 0x157CCD6618539CA0 dt: 0x00000000032FFF30 tag: 3 type: 0
 ***            ADC data: block:   898 ptr: 10599 data: 0xC2006F1C module:  8 fail: 0 range: 0 id:   512 channel:  0 adc: 28444  ts: 0x157CCD661A09CCCA OLD: ts: 0x157CCD661C0FA610 dt: 0x000000000205D946 tag: 3 type: 0
 ***            ADC data: block:   899 ptr:  6563 data: 0xC2246CFE module:  8 fail: 0 range: 0 id:   548 channel: 36 adc: 27902  ts: 0x157CCD6612D9B4DA OLD: ts: 0x157CCD661FC45A4A dt: 0x000000000CEAA570 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   900 ptr: 11403 data: 0x834CD425 module:  3 information type:  4 information field: 0x000CD425 ts: 0x157CCD425E000000 OLD: ts: 0x157CCD6B3E000000 dt: 0x00000028E0000000 tag: 2 type: 5
 ***            ADC data: block:   900 ptr: 15007 data: 0xC0F69130 module:  3 fail: 0 range: 0 id:   246 channel: 54 adc: 37168  ts: 0x157CCD4254622920 OLD: ts: 0x157CCD425FC26D10 dt: 0x000000000B6043F0 tag: 3 type: 0
 ***            ADC data: block:   909 ptr:  6669 data: 0xC2836E3F module: 10 fail: 0 range: 0 id:   643 channel:  3 adc: 28223  ts: 0x157CCD6B602DCDB6 OLD: ts: 0x157CCD6B6988D002 dt: 0x00000000095B024C tag: 3 type: 0
 ***            ADC data: block:   915 ptr: 16011 data: 0xC1B06E38 module:  6 fail: 0 range: 0 id:   432 channel: 48 adc: 28216  ts: 0x157CCD6B7B817E6E OLD: ts: 0x157CCD6B7B818026 dt: 0x00000000000001B8 tag: 3 type: 0
 ***            ADC data: block:   970 ptr: 12317 data: 0xC052911C module:  1 fail: 0 range: 0 id:    82 channel: 18 adc: 37148  ts: 0x157CCD6C094B08F2 OLD: ts: 0x157CCD6C0C516F62 dt: 0x0000000003066670 tag: 3 type: 0
 ***            ADC data: block:   970 ptr: 15371 data: 0xC040903B module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 36923  ts: 0x157CCD6C0AB98542 OLD: ts: 0x157CCD6C0C517606 dt: 0x000000000197F0C4 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   971 ptr:  6041 data: 0x814CD6BE module:  1 information type:  4 information field: 0x000CD6BE ts: 0x157CCD6BEE000000 OLD: ts: 0x157CCD6C0E000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:   971 ptr: 10293 data: 0xC06691A5 module:  1 fail: 0 range: 0 id:   102 channel: 38 adc: 37285  ts: 0x157CCD6BEC516F62 OLD: ts: 0x157CCD6BEFF9CA12 dt: 0x0000000003A85AB0 tag: 3 type: 0
 ***            ADC data: block:   984 ptr: 11167 data: 0xC0306F03 module:  0 fail: 0 range: 0 id:    48 channel: 48 adc: 28419  ts: 0x157CCD6C35AAA61C OLD: ts: 0x157CCD6C35AAAAC2 dt: 0x00000000000004A6 tag: 3 type: 0
 ***            ADC data: block:   986 ptr:  8705 data: 0xC1E69003 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 36867  ts: 0x157CCD6C3BC8CDEC OLD: ts: 0x157CCD6C3ECB8A22 dt: 0x000000000302BC36 tag: 2 type: 3
 ***   SYNC100 timestamp: block:   986 ptr:  9167 data: 0x874CD67A module:  7 information type:  4 information field: 0x000CD67A ts: 0x157CCD67AC000000 OLD: ts: 0x157CCD6C3C000000 dt: 0x0000000490000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   990 ptr:  2647 data: 0x864CD637 module:  6 information type:  4 information field: 0x000CD637 ts: 0x157CCD637E000000 OLD: ts: 0x157CCD6C4E000000 dt: 0x00000008D0000000 tag: 2 type: 5
 ***            ADC data: block:   990 ptr:  5839 data: 0xC1A16A8E module:  6 fail: 0 range: 0 id:   417 channel: 33 adc: 27278  ts: 0x157CCD63754CAC6E OLD: ts: 0x157CCD637F7A297E dt: 0x000000000A2D7D10 tag: 3 type: 0
 ***            ADC data: block:  1002 ptr:   565 data: 0xC1026D91 module:  4 fail: 0 range: 0 id:   258 channel:  2 adc: 28049  ts: 0x157CCD6C6017A0B2 OLD: ts: 0x157CCD6C69755D52 dt: 0x00000000095DBCA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1002 ptr: 11911 data: 0x844CD680 module:  4 information type:  4 information field: 0x000CD680 ts: 0x157CCD6806000000 OLD: ts: 0x157CCD6C66000000 dt: 0x0000000460000000 tag: 2 type: 5
 ***            ADC data: block:  1008 ptr:  3409 data: 0xC1C5918F module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 37263  ts: 0x157CCD6C7370F3CC OLD: ts: 0x157CCD6C7DCE4F8E dt: 0x000000000A5D5BC2 tag: 3 type: 0
 ***            ADC data: block:  1009 ptr:  1383 data: 0xC1E3907C module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 36988  ts: 0x157CCD6C750E896C OLD: ts: 0x157CCD6C7A3E319C dt: 0x00000000052FA830 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1010 ptr:  3937 data: 0x874CD447 module:  7 information type:  4 information field: 0x000CD447 ts: 0x157CCD4470000000 OLD: ts: 0x157CCD6C70000000 dt: 0x0000002800000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1014 ptr: 14811 data: 0x804CD6C3 module:  0 information type:  4 information field: 0x000CD6C3 ts: 0x157CCD6C3E000000 OLD: ts: 0x157CCD6C9E000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  1020 ptr:  3017 data: 0xC1AF6C63 module:  6 fail: 0 range: 0 id:   431 channel: 47 adc: 27747  ts: 0x157CCD6CB01B33EE OLD: ts: 0x157CCD6CB64028EC dt: 0x000000000624F4FE tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1023 ptr:  7423 data: 0x804CD6CB module:  0 information type:  4 information field: 0x000CD6CB ts: 0x157CCD6CB00B71FC OLD: ts: 0x157CCD6CC00B71FC dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1024 ptr:  4205 data: 0xC03D636B module:  0 fail: 0 range: 0 id:    61 channel: 61 adc: 25451  ts: 0x157CCD6CB5776C6C OLD: ts: 0x157CCD6CB64028EC dt: 0x0000000000C8BC80 tag: 3 type: 0
 ***            ADC data: block:  1024 ptr: 14635 data: 0xC1876FC7 module:  6 fail: 0 range: 0 id:   391 channel:  7 adc: 28615  ts: 0x157CCD6CC0FC618E OLD: ts: 0x157CCD6CCA2E276E dt: 0x000000000931C5E0 tag: 2 type: 3
 ***   SYNC100 timestamp: block:  1025 ptr:  7729 data: 0x864CD6C6 module:  6 information type:  4 information field: 0x000CD6C6 ts: 0x157CCD6C66000000 OLD: ts: 0x157CCD6CC6000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  1026 ptr:  9473 data: 0xC1D59163 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 37219  ts: 0x157CCD6C6D577ECC OLD: ts: 0x157CCD6C6D57819C dt: 0x00000000000002D0 tag: 3 type: 0
 ***            ADC data: block:  1041 ptr:  3423 data: 0xC1206D48 module:  4 fail: 0 range: 0 id:   288 channel: 32 adc: 27976  ts: 0x157CCD6CF6808CF2 OLD: ts: 0x157CCD6CFE53F5D2 dt: 0x0000000007D368E0 tag: 3 type: 0
 ***            ADC data: block:  1041 ptr: 12221 data: 0xC10F6D6B module:  4 fail: 0 range: 0 id:   271 channel: 15 adc: 28011  ts: 0x157CCD6CF7E053A2 OLD: ts: 0x157CCD6CF7EF16EE dt: 0x00000000000EC34C tag: 3 type: 0
 ***            ADC data: block:  1057 ptr:  7045 data: 0xC1BF6C99 module:  6 fail: 0 range: 0 id:   447 channel: 63 adc: 27801  ts: 0x157CCD6D3F02264E OLD: ts: 0x157CCD6D3F10F0AC dt: 0x00000000000ECA5E tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1059 ptr:  6413 data: 0x864CD6CE module:  6 information type:  4 information field: 0x000CD6CE ts: 0x157CCD6CE00456EE OLD: ts: 0x157CCD6D400456EE dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  1060 ptr:  3171 data: 0xC1B06E32 module:  6 fail: 0 range: 0 id:   432 channel: 48 adc: 28210  ts: 0x157CCD6CE46FA53E OLD: ts: 0x157CCD6CE61AA0CE dt: 0x0000000001AAFB90 tag: 3 type: 0
 ***            ADC data: block:  1069 ptr: 11411 data: 0xC1DD911A module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 37146  ts: 0x157CCD6D689875BC OLD: ts: 0x157CCD6D6898788C dt: 0x00000000000002D0 tag: 3 type: 0
 ***            ADC data: block:  1070 ptr: 16055 data: 0xC1FE9221 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 37409  ts: 0x157CCD6D6330B7DC OLD: ts: 0x157CCD6D6DE7E55C dt: 0x000000000AB72D80 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1071 ptr:  1331 data: 0x874CD6CD module:  7 information type:  4 information field: 0x000CD6CD ts: 0x157CCD6CD4000000 OLD: ts: 0x157CCD6D64000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  1080 ptr: 14349 data: 0xC1345480 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 21632  ts: 0x157CCD6D9661DB82 OLD: ts: 0x157CCD6D97CA5E22 dt: 0x00000000016882A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1081 ptr:  8787 data: 0x844CD4FE module:  4 information type:  4 information field: 0x000CD4FE ts: 0x157CCD4FEC000000 OLD: ts: 0x157CCD6D9C000000 dt: 0x0000001DB0000000 tag: 2 type: 5
 ***            ADC data: block:  1081 ptr: 12325 data: 0xC1439022 module:  5 fail: 0 range: 0 id:   323 channel:  3 adc: 36898  ts: 0x157CCD4FE7CA600C OLD: ts: 0x157CCD4FEDE65B62 dt: 0x00000000061BFB56 tag: 3 type: 0
 ***            ADC data: block:  1089 ptr: 13865 data: 0xC0366E3F module:  0 fail: 0 range: 0 id:    54 channel: 54 adc: 28223  ts: 0x157CCD6DB08D5E8C OLD: ts: 0x157CCD6DB08D638C dt: 0x0000000000000500 tag: 3 type: 0
 ***            ADC data: block:  1100 ptr:  4091 data: 0xC17E90CB module:  5 fail: 0 range: 0 id:   382 channel: 62 adc: 37067  ts: 0x157CCD6D805B262C OLD: ts: 0x157CCD6D8BDBBFAC dt: 0x000000000B809980 tag: 3 type: 0
 ***            ADC data: block:  1120 ptr: 11877 data: 0xC26D910A module:  9 fail: 0 range: 0 id:   621 channel: 45 adc: 37130  ts: 0x157CCD6E16C31386 OLD: ts: 0x157CCD6E16C317F0 dt: 0x000000000000046A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1130 ptr:  3755 data: 0x834CD6B6 module:  3 information type:  4 information field: 0x000CD6B6 ts: 0x157CCD6B60000610 OLD: ts: 0x157CCD6E30000610 dt: 0x00000002D0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1136 ptr: 10179 data: 0x854CD6DE module:  5 information type:  4 information field: 0x000CD6DE ts: 0x157CCD6DE00BEF0C OLD: ts: 0x157CCD6E400BEF0C dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  1137 ptr:  3309 data: 0xC15E92A4 module:  5 fail: 0 range: 0 id:   350 channel: 30 adc: 37540  ts: 0x157CCD6DE240971C OLD: ts: 0x157CCD6DE4765DAC dt: 0x000000000235C690 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1138 ptr:  2031 data: 0x854CD63B module:  5 information type:  4 information field: 0x000CD63B ts: 0x157CCD63BA000000 OLD: ts: 0x157CCD6DEA000000 dt: 0x0000000A30000000 tag: 2 type: 5
 ***            ADC data: block:  1141 ptr:  1071 data: 0xC16491EE module:  5 fail: 0 range: 0 id:   356 channel: 36 adc: 37358  ts: 0x157CCD6E5055057C OLD: ts: 0x157CCD6E5361D7DC dt: 0x00000000030CD260 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1141 ptr:  8973 data: 0x854CD63C module:  5 information type:  4 information field: 0x000CD63C ts: 0x157CCD63C4000000 OLD: ts: 0x157CCD6E54000000 dt: 0x0000000A90000000 tag: 2 type: 5
 ***            ADC data: block:  1141 ptr: 15419 data: 0xC17394F7 module:  5 fail: 0 range: 0 id:   371 channel: 51 adc: 38135  ts: 0x157CCD63C361D7DC OLD: ts: 0x157CCD63C703BECC dt: 0x0000000003A1E6F0 tag: 3 type: 0
 ***            ADC data: block:  1150 ptr:  7721 data: 0xC0E69170 module:  3 fail: 0 range: 0 id:   230 channel: 38 adc: 37232  ts: 0x157CCD6E732024C0 OLD: ts: 0x157CCD6E73202948 dt: 0x0000000000000488 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1151 ptr: 12549 data: 0x8B4CD6E3 module: 11 information type:  4 information field: 0x000CD6E3 ts: 0x157CCD6E3C000000 OLD: ts: 0x157CCD6E7C000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  1152 ptr:  2891 data: 0xC0C89057 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 36951  ts: 0x157CCD6E3649B6C0 OLD: ts: 0x157CCD6E3F27D708 dt: 0x0000000008DE2048 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1154 ptr:  6777 data: 0x8B4CD6E7 module: 11 information type:  4 information field: 0x000CD6E7 ts: 0x157CCD6E78000000 OLD: ts: 0x157CCD6E88000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1155 ptr:  1091 data: 0xC24C90B3 module:  9 fail: 0 range: 0 id:   588 channel: 12 adc: 37043  ts: 0x157CCD6E73A77316 OLD: ts: 0x157CCD6E7CF87C68 dt: 0x0000000009510952 tag: 3 type: 0
 ***            ADC data: block:  1163 ptr: 11171 data: 0xC2549039 module:  9 fail: 0 range: 0 id:   596 channel: 20 adc: 36921  ts: 0x157CCD6EA5D047B6 OLD: ts: 0x157CCD6EADF9CBE6 dt: 0x0000000008298430 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1163 ptr: 11651 data: 0x894CD423 module:  9 information type:  4 information field: 0x000CD423 ts: 0x157CCD4236000000 OLD: ts: 0x157CCD6EA6000000 dt: 0x0000002C70000000 tag: 2 type: 5
 ***            ADC data: block:  1164 ptr:  9147 data: 0xC276915F module:  9 fail: 0 range: 0 id:   630 channel: 54 adc: 37215  ts: 0x157CCD42380CBDA6 OLD: ts: 0x157CCD423C7F2046 dt: 0x00000000047262A0 tag: 3 type: 0
 ***            ADC data: block:  1168 ptr: 11481 data: 0xC25C900A module:  9 fail: 0 range: 0 id:   604 channel: 28 adc: 36874  ts: 0x157CCD6EB2C800A6 OLD: ts: 0x157CCD6EBA3FCD88 dt: 0x000000000777CCE2 tag: 3 type: 0
 ***            ADC data: block:  1170 ptr:  6445 data: 0xC2CB9363 module: 11 fail: 0 range: 0 id:   715 channel: 11 adc: 37731  ts: 0x157CCD6EC12B8808 OLD: ts: 0x157CCD6ECAFE0E18 dt: 0x0000000009D28610 tag: 3 type: 0
 ***            ADC data: block:  1192 ptr: 12145 data: 0xC07798B9 module:  1 fail: 0 range: 0 id:   119 channel: 55 adc: 39097  ts: 0x157CCD6F14529912 OLD: ts: 0x157CCD6F19C67982 dt: 0x000000000573E070 tag: 3 type: 0
 ***            ADC data: block:  1193 ptr: 10121 data: 0xC07394F3 module:  1 fail: 0 range: 0 id:   115 channel: 51 adc: 38131  ts: 0x157CCD6F19C67982 OLD: ts: 0x157CCD6F1B018912 dt: 0x00000000013B0F90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1246 ptr:  2499 data: 0x824CD6F1 module:  2 information type:  4 information field: 0x000CD6F1 ts: 0x157CCD6F18000000 OLD: ts: 0x157CCD6F88000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block:  1316 ptr:  8673 data: 0xC2146F7A module:  8 fail: 0 range: 0 id:   532 channel: 20 adc: 28538  ts: 0x157CCD6FFE99192A OLD: ts: 0x157CCD6FFEC6D446 dt: 0x00000000002DBB1C tag: 3 type: 0
 ***            ADC data: block:  1330 ptr:  2751 data: 0xC2A06E53 module: 10 fail: 0 range: 0 id:   672 channel: 32 adc: 28243  ts: 0x157CCD701AA411A6 OLD: ts: 0x157CCD701AA412D2 dt: 0x000000000000012C tag: 3 type: 0
 ***            ADC data: block:  1382 ptr:  3949 data: 0xC2026D6D module:  8 fail: 0 range: 0 id:   514 channel:  2 adc: 28013  ts: 0x157CCD70A3B2345A OLD: ts: 0x157CCD70AC2DA142 dt: 0x00000000087B6CE8 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1431 ptr: 14173 data: 0x864CD70C module:  6 information type:  4 information field: 0x000CD70C ts: 0x157CCD70C6000000 OLD: ts: 0x157CCD7106000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  1432 ptr:  6507 data: 0xC1B56D5E module:  6 fail: 0 range: 0 id:   437 channel: 53 adc: 27998  ts: 0x157CCD70C122882E OLD: ts: 0x157CCD70CA00F9DE dt: 0x0000000008DE71B0 tag: 3 type: 0
 ***            ADC data: block:  1438 ptr:    51 data: 0xC2866B6C module: 10 fail: 0 range: 0 id:   646 channel:  6 adc: 27500  ts: 0x157CCD7110A5F966 OLD: ts: 0x157CCD711214BC42 dt: 0x00000000016EC2DC tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1439 ptr:  2773 data: 0x8A4CD6B4 module: 10 information type:  4 information field: 0x000CD6B4 ts: 0x157CCD6B40049696 OLD: ts: 0x157CCD7110049696 dt: 0x00000005D0000000 tag: 2 type: 5
 ***            ADC data: block:  1440 ptr: 14757 data: 0xC2926E91 module: 10 fail: 0 range: 0 id:   658 channel: 18 adc: 28305  ts: 0x157CCD6B43CEE806 OLD: ts: 0x157CCD6B4D347926 dt: 0x0000000009659120 tag: 3 type: 0
 ***            ADC data: block:  1441 ptr: 12733 data: 0xC2A36F0D module: 10 fail: 0 range: 0 id:   675 channel: 35 adc: 28429  ts: 0x157CCD6B456BDE16 OLD: ts: 0x157CCD6B4A7DB0F6 dt: 0x000000000511D2E0 tag: 3 type: 0
 ***            ADC data: block:  1476 ptr:   279 data: 0xC1036E83 module:  4 fail: 0 range: 0 id:   259 channel:  3 adc: 28291  ts: 0x157CCD7093B23892 OLD: ts: 0x157CCD709400FA82 dt: 0x00000000004EC1F0 tag: 3 type: 0
 ***            ADC data: block:  1476 ptr: 14629 data: 0xC1306EBA module:  4 fail: 0 range: 0 id:   304 channel: 48 adc: 28346  ts: 0x157CCD70A6C63D92 OLD: ts: 0x157CCD70AB36C042 dt: 0x00000000047082B0 tag: 3 type: 0
 ***            ADC data: block:  1477 ptr: 12605 data: 0xC12B6BC7 module:  4 fail: 0 range: 0 id:   299 channel: 43 adc: 27591  ts: 0x157CCD70A79DC372 OLD: ts: 0x157CCD70AE3BE892 dt: 0x00000000069E2520 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1484 ptr:  6151 data: 0x804CD6CD module:  0 information type:  4 information field: 0x000CD6CD ts: 0x157CCD6CD4000000 OLD: ts: 0x157CCD71B4000000 dt: 0x00000004E0000000 tag: 2 type: 5
 ***            ADC data: block:  1485 ptr:  2175 data: 0xC0246C7A module:  0 fail: 0 range: 0 id:    36 channel: 36 adc: 27770  ts: 0x157CCD6CD117EE7C OLD: ts: 0x157CCD6CD9DF357C dt: 0x0000000008C74700 tag: 3 type: 0
 ***            ADC data: block:  1504 ptr:  3385 data: 0xC1C09064 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 36964  ts: 0x157CCD7204336E9C OLD: ts: 0x157CCD720853F3AC dt: 0x0000000004208510 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1504 ptr:  7291 data: 0x874CD6DA module:  7 information type:  4 information field: 0x000CD6DA ts: 0x157CCD6DA6000000 OLD: ts: 0x157CCD7206000000 dt: 0x0000000460000000 tag: 2 type: 5
 ***            ADC data: block:  1505 ptr:  1361 data: 0xC1FA93B9 module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 37817  ts: 0x157CCD6DA853F3AC OLD: ts: 0x157CCD6DAAF1D78C dt: 0x00000000029DE3E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1521 ptr: 16037 data: 0x854CD71F module:  5 information type:  4 information field: 0x000CD71F ts: 0x157CCD71FC000000 OLD: ts: 0x157CCD723C000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  1522 ptr:  6575 data: 0xC1CB948C module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 38028  ts: 0x157CCD71F654CEAC OLD: ts: 0x157CCD71FF330B3C dt: 0x0000000008DE3C90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1526 ptr: 12383 data: 0x854CD723 module:  5 information type:  4 information field: 0x000CD723 ts: 0x157CCD7238000000 OLD: ts: 0x157CCD7248000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1527 ptr:  6859 data: 0xC0CC902D module:  3 fail: 0 range: 0 id:   204 channel: 12 adc: 36909  ts: 0x157CCD7233B266E0 OLD: ts: 0x157CCD723D03992C dt: 0x000000000951324C tag: 3 type: 0
 ***            ADC data: block:  1533 ptr:  6971 data: 0xC0DC910E module:  3 fail: 0 range: 0 id:   220 channel: 28 adc: 37134  ts: 0x157CCD72554C7990 OLD: ts: 0x157CCD72554C7B0C dt: 0x000000000000017C tag: 3 type: 0
 ***            ADC data: block:  1557 ptr:  2075 data: 0xC2CF91A4 module: 11 fail: 0 range: 0 id:   719 channel: 15 adc: 37284  ts: 0x157CCD728CA8B018 OLD: ts: 0x157CCD728FE48DB8 dt: 0x00000000033BDDA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1557 ptr:  9221 data: 0x8B4CD6F0 module: 11 information type:  4 information field: 0x000CD6F0 ts: 0x157CCD6F00000108 OLD: ts: 0x157CCD7280000108 dt: 0x0000000380000000 tag: 2 type: 5
 ***            ADC data: block:  1593 ptr: 15703 data: 0xC26992FE module:  9 fail: 0 range: 0 id:   617 channel: 41 adc: 37630  ts: 0x157CCD72D56A66F6 OLD: ts: 0x157CCD72DCEC0FE6 dt: 0x000000000781A8F0 tag: 3 type: 0
 ***            ADC data: block:  1634 ptr: 15037 data: 0xC07992CD module:  1 fail: 0 range: 0 id:   121 channel: 57 adc: 37581  ts: 0x157CCD7317CC41E2 OLD: ts: 0x157CCD731E8E4D92 dt: 0x0000000006C20BB0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1635 ptr:  7677 data: 0x814CD6F2 module:  1 information type:  4 information field: 0x000CD6F2 ts: 0x157CCD6F2C000000 OLD: ts: 0x157CCD731C000000 dt: 0x00000003F0000000 tag: 2 type: 5
 ***            ADC data: block:  1682 ptr:  2585 data: 0xC0589233 module:  1 fail: 0 range: 0 id:    88 channel: 24 adc: 37427  ts: 0x157CCD738604F782 OLD: ts: 0x157CCD738604F9F8 dt: 0x0000000000000276 tag: 3 type: 0
 ***            ADC data: block:  1685 ptr:  1185 data: 0xC0A86CD8 module:  2 fail: 0 range: 0 id:   168 channel: 40 adc: 27864  ts: 0x157CCD7384459170 OLD: ts: 0x157CCD738D899CB0 dt: 0x0000000009440B40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1685 ptr: 13495 data: 0x824CD6F9 module:  2 information type:  4 information field: 0x000CD6F9 ts: 0x157CCD6F9A000000 OLD: ts: 0x157CCD738A000000 dt: 0x00000003F0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1691 ptr: 12193 data: 0x8B4CD72A module: 11 information type:  4 information field: 0x000CD72A ts: 0x157CCD72A0004598 OLD: ts: 0x157CCD73B0004598 dt: 0x0000000110000000 tag: 2 type: 5
 ***            ADC data: block:  1692 ptr:  4421 data: 0xC2F99424 module: 11 fail: 0 range: 0 id:   761 channel: 57 adc: 37924  ts: 0x157CCD72A2B5F498 OLD: ts: 0x157CCD72A3FFC538 dt: 0x000000000149D0A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1704 ptr:  1123 data: 0x834CD6E5 module:  3 information type:  4 information field: 0x000CD6E5 ts: 0x157CCD6E50000000 OLD: ts: 0x157CCD73E0000000 dt: 0x0000000590000000 tag: 2 type: 5
 ***            ADC data: block:  1715 ptr: 10963 data: 0xC22A6E6B module:  8 fail: 0 range: 0 id:   554 channel: 42 adc: 28267  ts: 0x157CCD74039CB81A OLD: ts: 0x157CCD74039EE86A dt: 0x0000000000023050 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1715 ptr: 11777 data: 0x884CD6B9 module:  8 information type:  4 information field: 0x000CD6B9 ts: 0x157CCD6B94000000 OLD: ts: 0x157CCD7404000000 dt: 0x0000000870000000 tag: 2 type: 5
 ***            ADC data: block:  1716 ptr:  8939 data: 0xC0CC9028 module:  3 fail: 0 range: 0 id:   204 channel: 12 adc: 36904  ts: 0x157CCD6B939EEB80 OLD: ts: 0x157CCD6B9A4B90AA dt: 0x0000000006ACA52A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1717 ptr:  3941 data: 0x834CD73F module:  3 information type:  4 information field: 0x000CD73F ts: 0x157CCD73F0000000 OLD: ts: 0x157CCD7400000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1718 ptr:  1407 data: 0xC2866B64 module: 10 fail: 0 range: 0 id:   646 channel:  6 adc: 27492  ts: 0x157CCD73F5781936 OLD: ts: 0x157CCD73F67E0CB0 dt: 0x000000000105F37A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1721 ptr: 13463 data: 0x884CD740 module:  8 information type:  4 information field: 0x000CD740 ts: 0x157CCD7404000000 OLD: ts: 0x157CCD7414000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1722 ptr: 10643 data: 0xC24F922C module:  9 fail: 0 range: 0 id:   591 channel: 15 adc: 37420  ts: 0x157CCD7400FCA006 OLD: ts: 0x157CCD740A4DC0FA dt: 0x00000000095120F4 tag: 3 type: 0
 ***            ADC data: block:  1724 ptr:    15 data: 0xC245919B module:  9 fail: 0 range: 0 id:   581 channel:  5 adc: 37275  ts: 0x157CCD7402D5D4A6 OLD: ts: 0x157CCD740C26F8B0 dt: 0x000000000951240A tag: 3 type: 0
 ***            ADC data: block:  1736 ptr:  6405 data: 0xC2936E60 module: 10 fail: 0 range: 0 id:   659 channel: 19 adc: 28256  ts: 0x157CCD742D0772C6 OLD: ts: 0x157CCD742D0773F2 dt: 0x000000000000012C tag: 3 type: 0
 ***            ADC data: block:  1736 ptr:  9451 data: 0xC2519301 module:  9 fail: 0 range: 0 id:   593 channel: 17 adc: 37633  ts: 0x157CCD742D16A7E6 OLD: ts: 0x157CCD742D726996 dt: 0x00000000005BC1B0 tag: 3 type: 0
 ***            ADC data: block:  1747 ptr: 14831 data: 0xC0729A65 module:  1 fail: 0 range: 0 id:   114 channel: 50 adc: 39525  ts: 0x157CCD7448893372 OLD: ts: 0x157CCD7448893930 dt: 0x00000000000005BE tag: 3 type: 0
 ***            ADC data: block:  1748 ptr: 14441 data: 0xC0D7914F module:  3 fail: 0 range: 0 id:   215 channel: 23 adc: 37199  ts: 0x157CCD744ADD1440 OLD: ts: 0x157CCD744D9BD0C2 dt: 0x0000000002BEBC82 tag: 3 type: 0
 ***            ADC data: block:  1754 ptr:  4429 data: 0xC0996B7C module:  2 fail: 0 range: 0 id:   153 channel: 25 adc: 27516  ts: 0x157CCD74768F1180 OLD: ts: 0x157CCD7479959730 dt: 0x00000000030685B0 tag: 3 type: 0
 ***            ADC data: block:  1755 ptr:  2405 data: 0xC09B6C96 module:  2 fail: 0 range: 0 id:   155 channel: 27 adc: 27798  ts: 0x157CCD7479959730 OLD: ts: 0x157CCD747D3DDA70 dt: 0x0000000003A84340 tag: 3 type: 0
 ***            ADC data: block:  1757 ptr:   415 data: 0xC09E6CF2 module:  2 fail: 0 range: 0 id:   158 channel: 30 adc: 27890  ts: 0x157CCD7486F34080 OLD: ts: 0x157CCD748E967B90 dt: 0x0000000007A33B10 tag: 3 type: 0
 ***            ADC data: block:  1759 ptr: 10721 data: 0xC0866BF4 module:  2 fail: 0 range: 0 id:   134 channel:  6 adc: 27636  ts: 0x157CCD74934A8FE0 OLD: ts: 0x157CCD749C02EA20 dt: 0x0000000008B85A40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1760 ptr:  5497 data: 0x8A4CD740 module: 10 information type:  4 information field: 0x000CD740 ts: 0x157CCD740E000000 OLD: ts: 0x157CCD74AE000000 dt: 0x00000000A0000000 tag: 2 type: 5
 ***            ADC data: block:  1764 ptr:  4371 data: 0xC0AB6B20 module:  2 fail: 0 range: 0 id:   171 channel: 43 adc: 27424  ts: 0x157CCD74B1E8F690 OLD: ts: 0x157CCD74B488FC24 dt: 0x0000000002A00594 tag: 2 type: 3
 ***            ADC data: block:  1778 ptr:  4999 data: 0xC2B56CFC module: 10 fail: 0 range: 0 id:   693 channel: 53 adc: 27900  ts: 0x157CCD74D1D98DD6 OLD: ts: 0x157CCD74DB8EB566 dt: 0x0000000009B52790 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1779 ptr:  1895 data: 0x8A4CD74B module: 10 information type:  4 information field: 0x000CD74B ts: 0x157CCD74B8000000 OLD: ts: 0x157CCD74D8000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  1798 ptr: 10245 data: 0xC01B6E2C module:  0 fail: 0 range: 0 id:    27 channel: 27 adc: 28204  ts: 0x157CCD75206A0A8C OLD: ts: 0x157CCD7520A6FFF0 dt: 0x00000000003CF564 tag: 3 type: 0
 ***            ADC data: block:  1801 ptr:  3615 data: 0xC00D6F77 module:  0 fail: 0 range: 0 id:    13 channel: 13 adc: 28535  ts: 0x157CCD7523B6DF8C OLD: ts: 0x157CCD7525B933DC dt: 0x0000000002025450 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1801 ptr: 12585 data: 0x804CD750 module:  0 information type:  4 information field: 0x000CD750 ts: 0x157CCD7508000000 OLD: ts: 0x157CCD7528000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  1802 ptr:  1591 data: 0xC2026D6D module:  8 fail: 0 range: 0 id:   514 channel:  2 adc: 28013  ts: 0x157CCD7505B9344A OLD: ts: 0x157CCD750A842CFC dt: 0x0000000004CAF8B2 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1819 ptr: 14941 data: 0x804CD752 module:  0 information type:  4 information field: 0x000CD752 ts: 0x157CCD7526000000 OLD: ts: 0x157CCD7556000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  1820 ptr:  6913 data: 0xC20D6E8F module:  8 fail: 0 range: 0 id:   525 channel: 13 adc: 28303  ts: 0x157CCD7544DCA5EA OLD: ts: 0x157CCD754E0F5FDC dt: 0x000000000932B9F2 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1823 ptr: 10107 data: 0x8A4CD754 module: 10 information type:  4 information field: 0x000CD754 ts: 0x157CCD7540095D66 OLD: ts: 0x157CCD7550095D66 dt: 0x0000000010000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1828 ptr: 11561 data: 0x804CD755 module:  0 information type:  4 information field: 0x000CD755 ts: 0x157CCD7558000000 OLD: ts: 0x157CCD7568000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1829 ptr:  2969 data: 0xC02E6CD1 module:  0 fail: 0 range: 0 id:    46 channel: 46 adc: 27857  ts: 0x157CCD75536BAAAC OLD: ts: 0x157CCD755BA9DB7C dt: 0x00000000083E30D0 tag: 3 type: 0
 ***            ADC data: block:  1834 ptr:  8965 data: 0xC22B6AFE module:  8 fail: 0 range: 0 id:   555 channel: 43 adc: 27390  ts: 0x157CCD757161E31A OLD: ts: 0x157CCD757161E680 dt: 0x0000000000000366 tag: 3 type: 0
 ***            ADC data: block:  1835 ptr: 10403 data: 0xC0996B77 module:  2 fail: 0 range: 0 id:   153 channel: 25 adc: 27511  ts: 0x157CCD7575ADA990 OLD: ts: 0x157CCD7575ADADFA dt: 0x000000000000046A tag: 3 type: 0
 ***            ADC data: block:  1836 ptr:  3227 data: 0xC21F6E68 module:  8 fail: 0 range: 0 id:   543 channel: 31 adc: 28264  ts: 0x157CCD7570FCA15A OLD: ts: 0x157CCD7577D37BFA dt: 0x0000000006D6DAA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1837 ptr:  1583 data: 0x884CD6AF module:  8 information type:  4 information field: 0x000CD6AF ts: 0x157CCD6AFE000000 OLD: ts: 0x157CCD757E000000 dt: 0x0000000A80000000 tag: 2 type: 5
 ***            ADC data: block:  1843 ptr: 10045 data: 0xC03A6CA8 module:  0 fail: 0 range: 0 id:    58 channel: 58 adc: 27816  ts: 0x157CCD7597FCAADC OLD: ts: 0x157CCD759E000000 dt: 0x0000000006035524 tag: 2 type: 4
 ***   SYNC100 timestamp: block:  1847 ptr: 12317 data: 0x844CD752 module:  4 information type:  4 information field: 0x000CD752 ts: 0x157CCD7526000000 OLD: ts: 0x157CCD75B6000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  1848 ptr: 10021 data: 0xC0056E90 module:  0 fail: 0 range: 0 id:     5 channel:  5 adc: 28304  ts: 0x157CCD752139BA1C OLD: ts: 0x157CCD752D5C8332 dt: 0x000000000C22C916 tag: 3 type: 0
 ***            ADC data: block:  1857 ptr:  5031 data: 0xC13A6CBE module:  4 fail: 0 range: 0 id:   314 channel: 58 adc: 27838  ts: 0x157CCD75D0DBCA32 OLD: ts: 0x157CCD75D87C6F52 dt: 0x0000000007A0A520 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1857 ptr: 14883 data: 0x844CD75C module:  4 information type:  4 information field: 0x000CD75C ts: 0x157CCD75C6000000 OLD: ts: 0x157CCD75D6000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1861 ptr:   663 data: 0xC01A6ECF module:  0 fail: 0 range: 0 id:    26 channel: 26 adc: 28367  ts: 0x157CCD75E46BAF1C OLD: ts: 0x157CCD75E46BB3C2 dt: 0x00000000000004A6 tag: 3 type: 0
 ***            ADC data: block:  1915 ptr: 11995 data: 0xC15D90E0 module:  5 fail: 0 range: 0 id:   349 channel: 29 adc: 37088  ts: 0x157CCD7673B269EC OLD: ts: 0x157CCD76792BC9AC dt: 0x0000000005795FC0 tag: 3 type: 0
 ***            ADC data: block:  1916 ptr:  9971 data: 0xC18E6CE5 module:  6 fail: 0 range: 0 id:   398 channel: 14 adc: 27877  ts: 0x157CCD76792BCD9E OLD: ts: 0x157CCD767A61427C dt: 0x00000000013574DE tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1922 ptr:  9033 data: 0x854CD767 module:  5 information type:  4 information field: 0x000CD767 ts: 0x157CCD767A000000 OLD: ts: 0x157CCD768A000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  1923 ptr:  5325 data: 0xC18F6DC9 module:  6 fail: 0 range: 0 id:   399 channel: 15 adc: 28105  ts: 0x157CCD767689674E OLD: ts: 0x157CCD767FDA929C dt: 0x0000000009512B4E tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1934 ptr: 14483 data: 0x834CD741 module:  3 information type:  4 information field: 0x000CD741 ts: 0x157CCD7418000000 OLD: ts: 0x157CCD76A8000000 dt: 0x0000000290000000 tag: 2 type: 5
 ***            ADC data: block:  1935 ptr:  1363 data: 0xC0F7983E module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 38974  ts: 0x157CCD7412B2D180 OLD: ts: 0x157CCD741984A200 dt: 0x0000000006D1D080 tag: 3 type: 0
 ***            ADC data: block:  1964 ptr:  7179 data: 0xC174AA51 module:  5 fail: 0 range: 0 id:   372 channel: 52 adc: 43601  ts: 0x157CCD76E7CAD95C OLD: ts: 0x157CCD76E7CADAD8 dt: 0x000000000000017C tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1975 ptr:  1169 data: 0x874CD722 module:  7 information type:  4 information field: 0x000CD722 ts: 0x157CCD722003BBCC OLD: ts: 0x157CCD76F003BBCC dt: 0x00000004D0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1976 ptr:  4823 data: 0x874CD6CD module:  7 information type:  4 information field: 0x000CD6CD ts: 0x157CCD6CDE000000 OLD: ts: 0x157CCD722E000000 dt: 0x0000000550000000 tag: 2 type: 5
 ***            ADC data: block:  1982 ptr: 14695 data: 0xC2CA9182 module: 11 fail: 0 range: 0 id:   714 channel: 10 adc: 37250  ts: 0x157CCD7703A29178 OLD: ts: 0x157CCD770821B978 dt: 0x00000000047F2800 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  1983 ptr: 11941 data: 0x8B4CD76C module: 11 information type:  4 information field: 0x000CD76C ts: 0x157CCD76CA000000 OLD: ts: 0x157CCD770A000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block:  1983 ptr: 12671 data: 0xC2D59296 module: 11 fail: 0 range: 0 id:   725 channel: 21 adc: 37526  ts: 0x157CCD76C821B978 OLD: ts: 0x157CCD76CA515298 dt: 0x00000000022F9920 tag: 3 type: 0
 ***            ADC data: block:  1996 ptr:  7875 data: 0xC2E4937A module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 37754  ts: 0x157CCD772A42D898 OLD: ts: 0x157CCD772A42DB0E dt: 0x0000000000000276 tag: 3 type: 0
 ***            ADC data: block:  1999 ptr: 16367 data: 0xC1A06D23 module:  6 fail: 0 range: 0 id:   416 channel: 32 adc: 27939  ts: 0x157CCD77312E958E OLD: ts: 0x157CCD77312E9782 dt: 0x00000000000001F4 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2003 ptr:  5333 data: 0x864CD711 module:  6 information type:  4 information field: 0x000CD711 ts: 0x157CCD71100D130E OLD: ts: 0x157CCD77300D130E dt: 0x0000000620000000 tag: 2 type: 5
 ***            ADC data: block:  2006 ptr:  7183 data: 0xC2499163 module:  9 fail: 0 range: 0 id:   585 channel:  9 adc: 37219  ts: 0x157CCD7763EB2BA6 OLD: ts: 0x157CCD776AA4220C dt: 0x0000000006B8F666 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2007 ptr:  1215 data: 0x894CD72E module:  9 information type:  4 information field: 0x000CD72E ts: 0x157CCD72EC000000 OLD: ts: 0x157CCD776C000000 dt: 0x0000000480000000 tag: 2 type: 5
 ***            ADC data: block:  2008 ptr:  1493 data: 0xC16D90B6 module:  5 fail: 0 range: 0 id:   365 channel: 45 adc: 37046  ts: 0x157CCD72FBCF4E2C OLD: ts: 0x157CCD72FD3515BC dt: 0x000000000165C790 tag: 3 type: 0
 ***            ADC data: block:  2016 ptr:  3509 data: 0xC2C38FF2 module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 36850  ts: 0x157CCD7781C88E18 OLD: ts: 0x157CCD778B19AC96 dt: 0x0000000009511E7E tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2023 ptr:  5389 data: 0x814CD733 module:  1 information type:  4 information field: 0x000CD733 ts: 0x157CCD733009A512 OLD: ts: 0x157CCD77A009A512 dt: 0x0000000470000000 tag: 2 type: 5
 ***            ADC data: block:  2023 ptr: 12413 data: 0xC247913B module:  9 fail: 0 range: 0 id:   583 channel:  7 adc: 37179  ts: 0x157CCD733029E2D6 OLD: ts: 0x157CCD733342CF42 dt: 0x000000000318EC6C tag: 3 type: 0
 ***            ADC data: block:  2025 ptr:  1859 data: 0xC2D19092 module: 11 fail: 0 range: 0 id:   721 channel: 17 adc: 37010  ts: 0x157CCD7333AEDC68 OLD: ts: 0x157CCD7333AEE12C dt: 0x00000000000004C4 tag: 3 type: 0
 ***            ADC data: block:  2027 ptr:  3689 data: 0xC2E89182 module: 11 fail: 0 range: 0 id:   744 channel: 40 adc: 37250  ts: 0x157CCD77A57F62C8 OLD: ts: 0x157CCD77AA20ABF8 dt: 0x0000000004A14930 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2027 ptr: 13351 data: 0x8B4CD771 module: 11 information type:  4 information field: 0x000CD771 ts: 0x157CCD771A000000 OLD: ts: 0x157CCD77AA000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block:  2028 ptr:  1665 data: 0xC04591D4 module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 37332  ts: 0x157CCD771A20B152 OLD: ts: 0x157CCD771C2E2BB8 dt: 0x00000000020D7A66 tag: 3 type: 0
 ***            ADC data: block:  2029 ptr: 12755 data: 0xC0599428 module:  1 fail: 0 range: 0 id:    89 channel: 25 adc: 37928  ts: 0x157CCD771029E952 OLD: ts: 0x157CCD771D878AD2 dt: 0x000000000D5DA180 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2038 ptr: 10673 data: 0x814CD77B module:  1 information type:  4 information field: 0x000CD77B ts: 0x157CCD77B0000000 OLD: ts: 0x157CCD77C0000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  2040 ptr:  8063 data: 0xC1CC90DC module:  7 fail: 0 range: 0 id:   460 channel: 12 adc: 37084  ts: 0x157CCD77C429702C OLD: ts: 0x157CCD77CE86978C dt: 0x000000000A5D2760 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2040 ptr: 11875 data: 0x874CD77A module:  7 information type:  4 information field: 0x000CD77A ts: 0x157CCD77A6000000 OLD: ts: 0x157CCD77C6000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  2044 ptr:  4419 data: 0xC1D49187 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 37255  ts: 0x157CCD77D2EE703C OLD: ts: 0x157CCD77DC211AFC dt: 0x000000000932AAC0 tag: 3 type: 0
 ***            ADC data: block:  2071 ptr:   975 data: 0xC1DE91C3 module:  7 fail: 0 range: 0 id:   478 channel: 30 adc: 37315  ts: 0x157CCD785FA30F5C OLD: ts: 0x157CCD785FA30FF2 dt: 0x0000000000000096 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  2073 ptr:  4985 data: 0x874CD77E module:  7 information type:  4 information field: 0x000CD77E ts: 0x157CCD77E001C7FC OLD: ts: 0x157CCD786001C7FC dt: 0x0000000080000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  2084 ptr: 13979 data: 0x814CD77D module:  1 information type:  4 information field: 0x000CD77D ts: 0x157CCD77D8000000 OLD: ts: 0x157CCD7898000000 dt: 0x00000000C0000000 tag: 2 type: 5
... 1137 more lines ...
Attachment 4: R4_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 ***            ADC data: block:   227 ptr:  4677 data: 0xC01B6E25 module:  0 fail: 0 range: 0 id:    27 channel: 27 adc: 28197  ts: 0x157CCF5D2D557EEA OLD: ts: 0x157CCF5D2D557F26 dt: 0x000000000000003C tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:        382
          ADC data format:    3125366 (  302390.4 Hz)
        Other data format:       1686 (     163.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:        843 (      81.6 Hz)
                              WR48-63:        843 (      81.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          1 (       0.1 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       10.336 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     279514        146          0          0          0          0         73         73          0          0          0          0
  1     276547        152          0          0          0          0         76         76          0          0          0          0
  2     272645        146          0          0          0          0         73         73          0          0          0          0
  3     270155        140          0          0          0          0         70         70          0          0          0          0
  4     266947        144          0          0          0          0         72         72          0          0          0          0
  5     255558        140          0          0          0          0         70         70          0          0          0          0
  6     260113        150          0          0          0          0         75         75          0          0          0          0
  7     245839        140          0          0          0          0         70         70          0          0          0          0
  8     254067        140          0          0          0          0         70         70          0          0          0          0
  9     251449        128          0          0          0          0         64         64          0          0          0          0
 10     247982        136          0          0          0          0         68         68          0          0          0          0
 11     244550        124          0          0          0          0         62         62          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          1          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 3507.020s (    0.109 blocks/s,   0.007 Mb/s)
Attachment 5: R7_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 ***   SYNC100 timestamp: block:   193 ptr:  5735 data: 0x8040C86B module:  0 information type:  4 information field: 0x0000C86B ts: 0x157D0C86BC000000 OLD: ts: 0x157D0C8E8C000000 dt: 0x00000007D0000000 tag: 2 type: 5
 *** ERROR: READ I/O error:       5002
                   blocks:       3990
          ADC data format:   28579740 (   45038.2 Hz)
        Other data format:    4082407 (    6433.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      68098 (     107.3 Hz)
                              WR48-63:      68098 (     107.3 Hz)
                           FEE64 disc:    3946211 (    6218.7 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:    3309837 (    5215.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      634.567 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.671
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2230947     483700          0          0          0          0       5688       5688     472324          0          0     470395
  1    2122270     348739          0          0          0          0       5674       5674     337391          0          0       3142
  2    2029271     265770          0          0          0          0       5674       5674     254422          0          0     296552
  3    1974300     348088          0          0          0          0       5674       5674     336740          0          0     156935
  4    2736837     381057          0          0          0          0       5670       5670     369717          0          0     574161
  5    4320720     341314          0          0          0          0       5674       5674     329966          0          0      40752
  6    2031325     266311          0          0          0          0       5674       5674     254963          0          0     570611
  7    2026551     356431          0          0          0          0       5674       5674     345083          0          0      34439
  8    2456505     344570          0          0          0          0       5674       5674     333222          0          0     457637
  9    2285340     347152          0          0          0          0       5674       5674     335804          0          0      14281
 10    2241663     250289          0          0          0          0       5674       5674     238941          0          0     680307
 11    2124011     348986          0          0          0          0       5674       5674     337638          0          0      10625
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          1          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 3522.477s (    1.133 blocks/s,   0.071 Mb/s)
Attachment 6: R10_0.out
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** FEE64 discriminator: block:     1 ptr:  9769 data: 0x8761FEDC module:  7 information type:  6 information field: 0x0001FEDC ts: 0x157CE748487E951C OLD: ts: 0x157CE7484E0B2A2E dt: 0x00000000058C9512 tag: 3 type: 0
 ***   SYNC100 timestamp: block:     2 ptr:  5647 data: 0x80411813 module:  0 information type:  4 information field: 0x00011813 ts: 0x157D118130000000 OLD: ts: 0x157DE74840000000 dt: 0x0000D5C710000000 tag: 2 type: 5
 *** ERROR: READ I/O error:       5002
                   blocks:       1639
          ADC data format:   11726597 (     251.0 Hz)
        Other data format:    1690263 (      36.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:         12 (       0.0 Hz)
                              SYNC100:      28591 (       0.6 Hz)
                              WR48-63:      28591 (       0.6 Hz)
                           FEE64 disc:    1633069 (      35.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:    1351746 (      28.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          1 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:    46723.492 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               29.863
  2                0.000               30.702
  3                0.000               29.696
  4                0.000               30.199
  5                0.000            46454.937
  6                0.000               30.031
  7                0.000               28.689
  8                0.000               30.367
  9                0.000               29.863
 10                0.000               29.828
 11                0.000               30.031
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     915119     202899          0          0          0          1       2382       2382     198134          0          0     190933
  1     873980     143527          0          0          0          1       2385       2385     138756          0          0       1342
  2     831172     109887          0          0          0          1       2375       2375     105136          0          0     121067
  3     812500     144292          0          0          0          1       2385       2385     139521          0          0      64139
  4    1124865     161438          0          0          0          1       2381       2381     156675          0          0     235472
  5    1756091     139805          0          0          0          1       2377       2377     135050          0          0      16138
  6     836702     109571          0          0          0          1       2383       2383     104804          0          0     233880
  7     834400     147902          0          0          0          1       2392       2392     143117          0          0      13899
  8    1008588     140605          0          0          0          1       2379       2379     135846          0          0     186267
  9     939402     143527          0          0          0          1       2385       2385     138756          0          0       5682
 10     922336     103917          0          0          0          1       2385       2385      99146          0          0     278712
 11     871442     142893          0          0          0          1       2382       2382     138128          0          0       4215
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          1          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          1          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 3534.123s (    0.464 blocks/s,   0.029 Mb/s)
Attachment 7: analyser.f
      PROGRAM GREAT

C     Read and analyse GREAT format data - version 3.3.0
C
C     See:
C     http://npg.dl.ac.uk/documents/edoc504/edoc504.html
C     http://npg.dl.ac.uk/DataAcq/TSformat.html

      IMPLICIT NONE                                 
C
C     External functions
C
      EXTERNAL timestamp
C
      DOUBLE PRECISION dfloat
C
      INTEGER and
      INTEGER*8 iargc
      INTEGER rshift
      INTEGER*8 timestamp
C
      REAL secnds
C
C     Parameter variables
C
      INTEGER block_length
      PARAMETER (block_length = 16384)
C
C     Local variables
C
      CHARACTER*255 arg
      CHARACTER*(block_length*4) buffer
      CHARACTER*255 filename
C
      INTEGER adc_data, block_number, block_ptr, ch, channel
      INTEGER channel_ident, data_block(block_length)
      INTEGER*8 dead(32)
      INTEGER event, fail
      INTEGER*8 first_ts
      INTEGER i
      INTEGER*2 i2(2), i2_2(4)
      INTEGER i4, i4_2(2)
      INTEGER*8 i8, idle(32)
      INTEGER information, information_index, io_error, items(0:32,0:31)
      INTEGER j, j4
      INTEGER*8 last_ts
      INTEGER mbs_data, module
      INTEGER*8 pause(32)
      INTEGER range
      INTEGER*8 resume(32)
      INTEGER sample_length, sum(0:31)
      INTEGER*8 sync(32)
      INTEGER tag, tag_old
      INTEGER*8 ts
      INTEGER ts28, ts48, ts64
      INTEGER*8 ts_old
      INTEGER type, type_old
C
      LOGICAL verbose
C
      REAL dt, t1, t2

      EQUIVALENCE ( data_block, buffer )
      EQUIVALENCE ( i4, i2( 1 ) )
      EQUIVALENCE ( i8, i4_2(1) )

      SAVE data_block, block_ptr, block_number, event

C----67---------------------------------------------------------------72------80
C     Announce program

      WRITE( 6, 9000 )

C     Initialise variables

      block_ptr = 1
      block_number = 0
      event = 0

      i4 = 0
      i8 = 0

      ts_old = 0
      tag_old = 0
      type_old = 0
      first_ts = 0
      last_ts = 0
      ts28 = 0
      ts48 = 0
      ts64 = 0

      type = 0
      
      verbose = .FALSE.
      
      DO i = 0, 32
       DO j = 0, 31
        sum( j ) = 0
        items( i, j ) = 0
       ENDDO
      ENDDO

      DO i = 1, 32
       pause( i ) = -1
       resume( i ) = -1
       dead( i ) = 0
       sync( i ) = 0
       idle( i ) = 0
      ENDDO

C     Get input filename from command line argument

      DO i = 1, iargc()
       CALL getarg( i, arg )
       IF ( arg.EQ.'v') THEN
        verbose = .TRUE.
        write(6,*) ' verbose'
       ELSE
        filename = arg
       ENDIF
      ENDDO
      IF ( iargc().LT.1 .OR. iargc().GT.2 ) THEN
       WRITE( 6, * ) ' Usage: ./a.exe [v] <filename>'
       STOP
      ENDIF

      t1 = SECNDS( 0.0 )

C     Open file

      OPEN( 10, RECL = 4 * block_length,
     +      FILE = filename, STATUS = 'OLD', ACCESS = 'DIRECT' )

    1 CONTINUE

C     Read next block
C     Note: for FORTRAN direct I/O first block/record number is 1
C           for GREAT data format first (encoded) block/record number is 0

      block_number = block_number + 1

      READ( 10, REC = block_number, iostat=io_error, ERR=1000 ) buffer

      DO block_ptr = 7, data_block( 6 ) / 4 + 7, 2


       i4 = data_block( block_ptr )
       j4 = data_block( block_ptr + 1 )

       IF ( i4.EQ.Z'ffffffff' ) THEN
        GOTO 1
       ENDIF

       IF ( i4.EQ.Z'5e5e5e5e' ) THEN
        GOTO 1
       ENDIF

       tag = AND( RSHIFT( i4, 30 ), Z'00000003' )

       module = 0
       type = 0

C      tag = 3 = ADC data format
C      tag = 2 = Other data
C      tag = 1 = Sample trace buffer format
C      tag = 0 = undefined?

C----67---------------------------------------------------------------72------80
C     ADC data

       IF ( tag.EQ.3 ) THEN

        fail = AND( RSHIFT( i4, 29 ), Z'00000001' )

        IF ( fail.NE.0 ) THEN

         WRITE( 6, 9600 ) block_number, block_ptr, i4

        ENDIF

        range = AND( RSHIFT( i4, 28 ), Z'00000001' )
        channel_ident = AND( RSHIFT( i4, 16 ), Z'00000fff' )
        module = AND( RSHIFT( channel_ident, 6 ), Z'0000003f' )
        IF ( range.EQ.1 ) THEN
         items(module, 19 ) = items(module, 19 ) + 1
        ENDIF
        channel = AND( channel_ident, Z'0000003f' )
        adc_data = AND( i4, Z'0000ffff' )

        items(module, 0 ) = items(module, 0 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        ts = timestamp( ts64, ts48, ts28, type )

	IF ( ts.LT.ts_old ) THEN

	 items(module, 20 ) = items(module, 20 ) + 1

         WRITE( 6, 9710 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts, ts_old, ts_old-ts,
     +   tag_old, type_old
        ENDIF

        ts_old = ts
        tag_old = tag
        type_old = 0

        IF ( verbose ) THEN 
         WRITE( 6, 9700 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts
        ENDIF

        ch = channel + ( module - 1 ) * 64 + ( range * 2048 )

C----67---------------------------------------------------------------72------80
C     Other information

       ELSEIF( tag.EQ.2 ) THEN

        module = AND( RSHIFT( i4, 24 ), Z'0000003f' )
        type = AND( RSHIFT( i4, 20 ), Z'0000000f' )
        information = AND( i4, Z'000fffff' )

        items(module, 1 ) = items(module, 1 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        IF ( type.EQ.2 ) THEN

C----67---------------------------------------------------------------72------80
C     PAUSE timestamp

         items(module, 4 ) = items(module, 4 ) + 1

        ts48 = information
C        ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 21 ) = items(module, 21 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         pause( module ) = ts

         IF ( verbose ) THEN 
          WRITE( 6, 9800 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.3 ) THEN

C----67---------------------------------------------------------------72------80
C     RESUME timestamp

         items(module, 5 ) = items(module, 5 ) + 1

         ts48 = information
C         ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 22 ) = items(module, 22 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         resume( module ) = ts

         IF ( resume( module ).GE.0 
     +         .AND.
     +        pause( module ).GE.0 
     +          .AND.
     +        resume( module ).GE.pause( module ) ) THEN
          dead( module ) = ( resume( module ) - pause( module ) ) 
     +                     + dead( module )
          pause( module ) = -1
          resume( module ) = -1
         ENDIF

         IF ( verbose ) THEN 
          WRITE( 6, 9810 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.4 ) THEN

C----67---------------------------------------------------------------72------80
C     SYNC100 timestamp/White Rabbit Timestamp Marker (bits 28-47)

         items(module, 6 ) = items(module, 6 ) + 1

... 457 more lines ...
  25   Fri Jan 25 12:33:03 2019 NHaida-gsi Access

If you have a GSI account it should be possible to connect to aida-gsi remotely via the following
SSH into lx-pool.gsi.de with GSI Linux username/password (ssh username@lx-pool.gsi.de)
Then SSH into aida-gsi via ssh npg@aida-3

  24   Fri Jan 25 12:26:17 2019 NHRaspberry Pi Startup & Information

Two raspberry pis:

nnrpi1 - FEE64 system consoles
nnrpi2 - AC Mains Relay & CAEN HV

Startup:

1. Plug in micro-USB on pis
2. Connect via ssh: ssh pi@nnrpi1  or ssh pi@nnrpi2
3. Start MIDAS:
cd /MIDAS/TclHttpd/Linux-arm
./TclHttpd-server

4. Connect from aida-gsi web browser
http://nnrpi2:8015 - For AC Relay Control
http://nnrpi1:8015 - For Pi Monitoring (Get list of USB terminals, Connect)
-> Parse USB log for details to check if all FEEs have finished booting completely or not

5. For CAEN HV connect via ssh/X (ssh -X pi@nnrpi2) and run
putty &
Opens putty window to connect to the CAEN HV module (Serial /dev/ttyACM0)

  23   Fri Jan 25 09:54:12 2019 CA, TD, NH, VP25th January 2019
09.30: AIDA still running, not writing to file
       Still running ok
       No water leaks

09.45: FEE64 temperatures ok (attachment 1)
       Good event statistics ok (attachment 2)

10.00: Results of System Wide Checks
       ADC Calibration check (attachment 3)
       Clock Status (attachment 4)
       Sync error counter (attachment 5)
       White Rabbit Decoder status (attachment 7)
       Memory Information (attachment 8) 
       
10.30: Clock ReSync performed (attachment 6)
       Start writing to file TapeData/NULL/R7_0

11.50: DAQ stopped, TapeServer stopped
       
       TapeServer restarted, DAQ restarted

       Writing to file TapeData/NULL/R9_0 with Clock ReSync at start.

       Wasn't writing to disk, restarting (DAQ stop, TapeServer stop, TapeServer start TO DISK, DAQ Start)

       Writing to file TapeData/NULL/R10_0

12.04: DAQ stopped, TapeServer stopped, merger stopped
       FEE64 power relay switched off
       Raspberry PIs switched off and disconnected
       water cooler stopped
       Mains IEC cables removed from relay

       
        
Attachment 1: 250119temp.png
250119temp.png
Attachment 2: 250119stats.png
250119stats.png
Attachment 3: adccalib.png
adccalib.png
Attachment 4: clockstatus.png
clockstatus.png
Attachment 5: syncerror.png
syncerror.png
Attachment 6: syncpulse.png
syncpulse.png
Attachment 7: wrdecoder.png
wrdecoder.png
Attachment 8: meminfo2.png
meminfo2.png
  22   Thu Jan 24 17:16:32 2019 CA, TD, NH, VPMACB time switch settings
AIDA@DESPEC MACB time switch settings before and after WR are as follows.

Attachment 1 - MACB 1-4 from left to right

MACB number:                           1        2        3        4

100MHZ clock setting (before WR):      0        2        2        2

White Rabbit setting (after WR):       3        3        3        3
Attachment 1: 50668223_286084358731841_9104059387265155072_n.jpg
50668223_286084358731841_9104059387265155072_n.jpg
Attachment 2: despecmacb.pdf
despecmacb.pdf
  21   Thu Jan 24 14:53:26 2019 TDAnalyses GSIJan19/R2_0 - R4_0
Attachment 1 - summary analysis R2_0

               increased slow comparator & LEC/MEC fast comparators cf. R1_0 to minimise random noise

Attachment 2 - summary analysis R3_0

               per R2_0 but BNC PB-5 pulser frequency increased from c. 2Hz to c. 1kHz
               pulser to *all* channels 12x FEE64s

Attachment 3 - summary analysis R4_0

               open TapeSever file, DAQ GO, Start ASIC Readout (*correct* DAQ start sequence)

Observations

- some RESUME but *zero* PAUSE data items observed

- information types 4 & 5 not equal for R2_0 and almost, but not quite, equal for R3_0 & R4_0 

- information types 4 & 5 start data file with 'correct' DAQ start sequence

           
Attachment 1: WR_sample_R2_0_24Jan19
[td@localhost COREYUSB]$ ~/analyser/analyser R2_0
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***            ADC data: block:     1 ptr:  5477 data: 0xC1E39CDA module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 40154  ts: 0x157CC966A64D2ABC OLD: ts: 0x157CC966AC9A969C dt: 0x00000000064D6BE0 tag: 3 type: 0
 ***            ADC data: block:    10 ptr:  7957 data: 0xC1FC80A3 module:  7 fail: 0 range: 0 id:   508 channel: 60 adc: 32931  ts: 0x157CC97ABC9A9E6C OLD: ts: 0x157CC97ABDE0AD9C dt: 0x0000000001460F30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    11 ptr:  5407 data: 0x874C966B module:  7 information type:  4 information field: 0x000C966B ts: 0x157CC966B00008BC OLD: ts: 0x157CC97AB00008BC dt: 0x0000001400000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    12 ptr: 12395 data: 0x874C960C module:  7 information type:  4 information field: 0x000C960C ts: 0x157CC960C6000000 OLD: ts: 0x157CC966B6000000 dt: 0x00000005F0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    24 ptr:  8551 data: 0x874C960D module:  7 information type:  4 information field: 0x000C960D ts: 0x157CC960D0000000 OLD: ts: 0x157CC97B30000000 dt: 0x0000001A60000000 tag: 2 type: 5
 ***            ADC data: block:    34 ptr: 14519 data: 0xC1F4813C module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33084  ts: 0x157CC97BA4A99ECC OLD: ts: 0x157CC97BAD06D89C dt: 0x00000000085D39D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    36 ptr:  4661 data: 0x874C960D module:  7 information type:  4 information field: 0x000C960D ts: 0x157CC960DA000000 OLD: ts: 0x157CC97BAA000000 dt: 0x0000001AD0000000 tag: 2 type: 5
 ***            ADC data: block:    47 ptr: 10505 data: 0xC1EC81A1 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33185  ts: 0x157CC97C21EEF58C OLD: ts: 0x157CC97C25D13D6C dt: 0x0000000003E247E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    48 ptr:  3189 data: 0x874C960E module:  7 information type:  4 information field: 0x000C960E ts: 0x157CC960E4000000 OLD: ts: 0x157CC97C24000000 dt: 0x0000001B40000000 tag: 2 type: 5
 ***            ADC data: block:    51 ptr:  4209 data: 0xC0439E5C module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 40540  ts: 0x157CC960E5D13E02 OLD: ts: 0x157CC960EF0DA1DC dt: 0x00000000093C63DA tag: 3 type: 0
 ***   SYNC100 timestamp: block:    60 ptr: 10519 data: 0x874C960F module:  7 information type:  4 information field: 0x000C960F ts: 0x157CC960F000105C OLD: ts: 0x157CC97C9000105C dt: 0x0000001BA0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    73 ptr: 14199 data: 0x874C9610 module:  7 information type:  4 information field: 0x000C9610 ts: 0x157CC961000029EC OLD: ts: 0x157CC97D100029EC dt: 0x0000001C10000000 tag: 2 type: 5
 ***            ADC data: block:    75 ptr: 11245 data: 0xC1D6A3DB module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 41947  ts: 0x157CC96104F56C9C OLD: ts: 0x157CC96106CDFC2C dt: 0x0000000001D88F90 tag: 3 type: 0
 ***            ADC data: block:    84 ptr: 13025 data: 0xC1F281C1 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33217  ts: 0x157CC97D86CDFC2C OLD: ts: 0x157CC97D8C89AAFC dt: 0x0000000005BBAED0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:    86 ptr:  2253 data: 0x874C9610 module:  7 information type:  4 information field: 0x000C9610 ts: 0x157CC9610C000000 OLD: ts: 0x157CC97D8C000000 dt: 0x0000001C80000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:    97 ptr: 13277 data: 0x874C9611 module:  7 information type:  4 information field: 0x000C9611 ts: 0x157CC96116000000 OLD: ts: 0x157CC97E06000000 dt: 0x0000001CF0000000 tag: 2 type: 5
 ***            ADC data: block:   110 ptr:  1203 data: 0xC1E97F8D module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32653  ts: 0x157CC97E7138489C OLD: ts: 0x157CC97E7A7A827C dt: 0x00000000094239E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   112 ptr:  5101 data: 0x874C9612 module:  7 information type:  4 information field: 0x000C9612 ts: 0x157CC9612A000000 OLD: ts: 0x157CC97E7A000000 dt: 0x0000001D50000000 tag: 2 type: 5
 ***            ADC data: block:   113 ptr: 11279 data: 0xC04EA04D module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 41037  ts: 0x157CC9612A7A8312 OLD: ts: 0x157CC9612F47020C dt: 0x0000000004CC7EFA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   123 ptr:  1653 data: 0x874C9613 module:  7 information type:  4 information field: 0x000C9613 ts: 0x157CC9613000016C OLD: ts: 0x157CC97EF000016C dt: 0x0000001DC0000000 tag: 2 type: 5
 ***            ADC data: block:   126 ptr:  7379 data: 0xC1E59FBB module:  7 fail: 0 range: 0 id:   485 channel: 37 adc: 40891  ts: 0x157CC961347DEC1C OLD: ts: 0x157CC9613CBC88BC dt: 0x00000000083E9CA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   135 ptr: 15431 data: 0x874C9613 module:  7 information type:  4 information field: 0x000C9613 ts: 0x157CC9613E000000 OLD: ts: 0x157CC97F6E000000 dt: 0x0000001E30000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   147 ptr: 12937 data: 0x874C9614 module:  7 information type:  4 information field: 0x000C9614 ts: 0x157CC96148000000 OLD: ts: 0x157CC97FE8000000 dt: 0x0000001EA0000000 tag: 2 type: 5
 ***            ADC data: block:   159 ptr: 16335 data: 0xC1C58102 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33026  ts: 0x157CC9805416AEFC OLD: ts: 0x157CC9805A0192CC dt: 0x0000000005EAE3D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   162 ptr:  2123 data: 0x874C9615 module:  7 information type:  4 information field: 0x000C9615 ts: 0x157CC9615C000000 OLD: ts: 0x157CC9805C000000 dt: 0x0000001F00000000 tag: 2 type: 5
 ***            ADC data: block:   172 ptr: 12809 data: 0xC1E18197 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33175  ts: 0x157CC980D1BFC0FC OLD: ts: 0x157CC980D2CDEB9C dt: 0x00000000010E2AA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   173 ptr: 15185 data: 0x874C9616 module:  7 information type:  4 information field: 0x000C9616 ts: 0x157CC96166000000 OLD: ts: 0x157CC980D6000000 dt: 0x0000001F70000000 tag: 2 type: 5
 ***            ADC data: block:   176 ptr:  4719 data: 0xC044A540 module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 42304  ts: 0x157CC96162CDEC32 OLD: ts: 0x157CC9616F2E90EC dt: 0x000000000C60A4BA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   185 ptr: 10317 data: 0x874C9617 module:  7 information type:  4 information field: 0x000C9617 ts: 0x157CC96170000000 OLD: ts: 0x157CC98140000000 dt: 0x0000001FD0000000 tag: 2 type: 5
 ***            ADC data: block:   188 ptr: 14841 data: 0xC04EA060 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 41056  ts: 0x157CC9617928FEB2 OLD: ts: 0x157CC9617C65DDEC dt: 0x00000000033CDF3A tag: 3 type: 0
 ***   SYNC100 timestamp: block:   199 ptr:  1305 data: 0x874C9618 module:  7 information type:  4 information field: 0x000C9618 ts: 0x157CC96180001BAC OLD: ts: 0x157CC981C0001BAC dt: 0x0000002040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   200 ptr:  2029 data: 0x874C95AE module:  7 information type:  4 information field: 0x000C95AE ts: 0x157CC95AE0000000 OLD: ts: 0x157CC96180000000 dt: 0x00000006A0000000 tag: 2 type: 5
 ***            ADC data: block:   200 ptr: 11715 data: 0xC1E59F81 module:  7 fail: 0 range: 0 id:   485 channel: 37 adc: 40833  ts: 0x157CC95AE1F629DC OLD: ts: 0x157CC95AE249229C dt: 0x000000000052F8C0 tag: 3 type: 0
 ***            ADC data: block:   209 ptr: 13579 data: 0xC1D58057 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32855  ts: 0x157CC98232492A6C OLD: ts: 0x157CC982398985AC dt: 0x0000000007405B40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   211 ptr: 15195 data: 0x874C95AE module:  7 information type:  4 information field: 0x000C95AE ts: 0x157CC95AEA000000 OLD: ts: 0x157CC9823A000000 dt: 0x0000002750000000 tag: 2 type: 5
 ***            ADC data: block:   213 ptr:  5489 data: 0xC1E2A092 module:  7 fail: 0 range: 0 id:   482 channel: 34 adc: 41106  ts: 0x157CC95AE98985AC OLD: ts: 0x157CC95AEF54568C dt: 0x0000000005CAD0E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   222 ptr: 11565 data: 0x874C95AF module:  7 information type:  4 information field: 0x000C95AF ts: 0x157CC95AF000A3EC OLD: ts: 0x157CC982B000A3EC dt: 0x00000027C0000000 tag: 2 type: 5
 ***            ADC data: block:   225 ptr:  2213 data: 0xC1E7A081 module:  7 fail: 0 range: 0 id:   487 channel: 39 adc: 41089  ts: 0x157CC95AF11CF11C OLD: ts: 0x157CC95AF8E659AC dt: 0x0000000007C96890 tag: 3 type: 0
 ***            ADC data: block:   234 ptr:  5573 data: 0xC1D2812A module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 33066  ts: 0x157CC98328E678EC OLD: ts: 0x157CC98329E9F5CC dt: 0x0000000001037CE0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   235 ptr: 12551 data: 0x874C95AF module:  7 information type:  4 information field: 0x000C95AF ts: 0x157CC95AFE000000 OLD: ts: 0x157CC9832E000000 dt: 0x0000002830000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   247 ptr: 10957 data: 0x874C95B0 module:  7 information type:  4 information field: 0x000C95B0 ts: 0x157CC95B08000000 OLD: ts: 0x157CC983A8000000 dt: 0x00000028A0000000 tag: 2 type: 5
 ***            ADC data: block:   250 ptr: 10065 data: 0xC045A02C module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 41004  ts: 0x157CC95B117BF2A2 OLD: ts: 0x157CC95B1363823C dt: 0x0000000001E78F9A tag: 3 type: 0
 ***            ADC data: block:   259 ptr: 11767 data: 0xC1C582BD module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33469  ts: 0x157CC9841363A17C OLD: ts: 0x157CC98417D82D6C dt: 0x0000000004748BF0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   262 ptr:  2993 data: 0x874C95B1 module:  7 information type:  4 information field: 0x000C95B1 ts: 0x157CC95B1C000000 OLD: ts: 0x157CC9841C000000 dt: 0x0000002900000000 tag: 2 type: 5
 ***            ADC data: block:   263 ptr:  3677 data: 0xC1FFA313 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 41747  ts: 0x157CC95B17D82D6C OLD: ts: 0x157CC95B1FF16FDC dt: 0x0000000008194270 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   272 ptr:  6217 data: 0x874C95B2 module:  7 information type:  4 information field: 0x000C95B2 ts: 0x157CC95B20005C2C OLD: ts: 0x157CC98490005C2C dt: 0x0000002970000000 tag: 2 type: 5
 ***            ADC data: block:   275 ptr:    55 data: 0xC0419F6F module:  1 fail: 0 range: 0 id:    65 channel:  1 adc: 40815  ts: 0x157CC95B20A3B3E2 OLD: ts: 0x157CC95B2A094C9C dt: 0x00000000096598BA tag: 3 type: 0
 ***   SYNC100 timestamp: block:   285 ptr: 12223 data: 0x874C95B3 module:  7 information type:  4 information field: 0x000C95B3 ts: 0x157CC95B30000000 OLD: ts: 0x157CC98500000000 dt: 0x00000029D0000000 tag: 2 type: 5
 ***            ADC data: block:   295 ptr: 15473 data: 0xC1C7819B module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 33179  ts: 0x157CC9857412AA6C OLD: ts: 0x157CC9857FCCF35C dt: 0x000000000BBA48F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   297 ptr:  9095 data: 0x874C95B3 module:  7 information type:  4 information field: 0x000C95B3 ts: 0x157CC95B3A000000 OLD: ts: 0x157CC9857A000000 dt: 0x0000002A40000000 tag: 2 type: 5
 ***            ADC data: block:   308 ptr: 11479 data: 0xC1D07F9E module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32670  ts: 0x157CC985F183FAFC OLD: ts: 0x157CC985F7602FEC dt: 0x0000000005DC34F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   309 ptr:  5215 data: 0x874C95B4 module:  7 information type:  4 information field: 0x000C95B4 ts: 0x157CC95B44000000 OLD: ts: 0x157CC985F4000000 dt: 0x0000002AB0000000 tag: 2 type: 5
 ***            ADC data: block:   312 ptr:  3389 data: 0xC045A028 module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 41000  ts: 0x157CC95B47603082 OLD: ts: 0x157CC95B4EAC24FC dt: 0x00000000074BF47A tag: 3 type: 0
 ***   SYNC100 timestamp: block:   321 ptr: 13307 data: 0x874C95B5 module:  7 information type:  4 information field: 0x000C95B5 ts: 0x157CC95B50005ABC OLD: ts: 0x157CC98670005ABC dt: 0x0000002B20000000 tag: 2 type: 5
 ***            ADC data: block:   324 ptr:  2051 data: 0xC1F09FD9 module:  7 fail: 0 range: 0 id:   496 channel: 48 adc: 40921  ts: 0x157CC95B5164E97C OLD: ts: 0x157CC95B591ED3BC dt: 0x0000000007B9EA40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   335 ptr:   493 data: 0x874C95B6 module:  7 information type:  4 information field: 0x000C95B6 ts: 0x157CC95B60000EBC OLD: ts: 0x157CC986E0000EBC dt: 0x0000002B80000000 tag: 2 type: 5
 *** FEE64 discriminator: block:   344 ptr: 14695 data: 0x87610020 module:  7 information type:  6 information field: 0x00010020 ts: 0x157CC9875292C7DC OLD: ts: 0x157CC9875E1A3B8C dt: 0x000000000B8773B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   347 ptr:  7031 data: 0x874C95B6 module:  7 information type:  4 information field: 0x000C95B6 ts: 0x157CC95B6C000000 OLD: ts: 0x157CC9875C000000 dt: 0x0000002BF0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   357 ptr:  9833 data: 0x874C95B7 module:  7 information type:  4 information field: 0x000C95B7 ts: 0x157CC95B7000284C OLD: ts: 0x157CC987D000284C dt: 0x0000002C60000000 tag: 2 type: 5
 ***            ADC data: block:   360 ptr:  2279 data: 0xC1D09D8B module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 40331  ts: 0x157CC95B75AD29FC OLD: ts: 0x157CC95B7962B6FC dt: 0x0000000003B58D00 tag: 3 type: 0
 ***            ADC data: block:   369 ptr:  5883 data: 0xC1C4800E module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32782  ts: 0x157CC9884962EDAC OLD: ts: 0x157CC9884D41BE4C dt: 0x0000000003DED0A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   371 ptr:  1713 data: 0x874C95B8 module:  7 information type:  4 information field: 0x000C95B8 ts: 0x157CC95B80000000 OLD: ts: 0x157CC98840000000 dt: 0x0000002CC0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:   382 ptr: 13055 data: 0x874C95B8 module:  7 information type:  4 information field: 0x000C95B8 ts: 0x157CC95B8A000000 OLD: ts: 0x157CC988CA000000 dt: 0x0000002D40000000 tag: 2 type: 5
 ***            ADC data: block:   949 ptr:     7 data: 0xC0AF5DD7 module:  2 fail: 0 range: 0 id:   175 channel: 47 adc: 24023  ts: 0x157CC9A593F069E0 OLD: ts: 0x157CC9A59CB632B0 dt: 0x0000000008C5C8D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:   949 ptr:   781 data: 0x824C9A52 module:  2 information type:  4 information field: 0x000C9A52 ts: 0x157CC9A52C000000 OLD: ts: 0x157CC9A59C000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  3022 ptr: 10885 data: 0x814CA0E0 module:  1 information type:  4 information field: 0x000CA0E0 ts: 0x157CCA0E00CE5EC2 OLD: ts: 0x157CCA0F80CE5EC2 dt: 0x0000000180000000 tag: 2 type: 5
 ***            ADC data: block:  8941 ptr:     7 data: 0xC0935FCF module:  2 fail: 0 range: 0 id:   147 channel: 19 adc: 24527  ts: 0x157CCB3DC32A17D0 OLD: ts: 0x157CCB3DCBE16200 dt: 0x0000000008B74A30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8941 ptr:   367 data: 0x824CB3D5 module:  2 information type:  4 information field: 0x000CB3D5 ts: 0x157CCB3D56000000 OLD: ts: 0x157CCB3DC6000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 11956 ptr:   211 data: 0x874CBD75 module:  7 information type:  4 information field: 0x000CBD75 ts: 0x157CCBD75041DF2C OLD: ts: 0x157CCBD7C041DF2C dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 13889 ptr:     7 data: 0xC1FFA34E module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 41806  ts: 0x157CCC3A6419FA8C OLD: ts: 0x157CCC3A6E09016C dt: 0x0000000009EF06E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 13889 ptr:  1019 data: 0x874CC39F module:  7 information type:  4 information field: 0x000CC39F ts: 0x157CCC39FE000000 OLD: ts: 0x157CCC3A6E000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 17602 ptr:     7 data: 0xC20F6111 module:  8 fail: 0 range: 0 id:   527 channel: 15 adc: 24849  ts: 0x157CCCF801D6514A OLD: ts: 0x157CCCF80A8EA51A dt: 0x0000000008B853D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 17602 ptr:   785 data: 0x884CCF79 module:  8 information type:  4 information field: 0x000CCF79 ts: 0x157CCCF79A000000 OLD: ts: 0x157CCCF80A000000 dt: 0x0000000070000000 tag: 2 type: 5
 *** ERROR: READ I/O error:       5002
                   blocks:      18050
          ADC data format:  146874482 (   36525.1 Hz)
        Other data format:     864819 (     215.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:     422370 (     105.0 Hz)
                              WR48-63:     422478 (     105.1 Hz)
                           FEE64 disc:      19971 (       5.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          5 (       0.0 Hz)

                Timewarps:        ADC:         33 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:         39 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          1 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     4021.191 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000            39433.647
  2                0.000            39406.302
  3                0.000            39369.351
  4                0.000            39369.339
  5                0.000            39369.359
  6                0.000            39369.359
  7                0.000            87766.281
  8                0.000            39387.796
  9                0.000            39369.386
 10                0.000            39369.415
 11                0.000            39369.346
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12179425      70380          0          0          0          0      35185      35195          0          0          0          0
  1   12189608      70368          0          0          0          0      35180      35188          0          0          0          0
  2   12206317      70449          0          0          0          0      35220      35229          0          0          0          0
  3   12187794      70373          0          0          0          0      35181      35192          0          0          0          0
  4   12181753      70374          0          0          0          0      35183      35191          0          0          0          0
  5   12182389      70382          0          0          0          0      35187      35195          0          0          0          0
  6   12191845      70387          0          0          0          0      35190      35197          0          0          0          0
  7   12807849      90535          0          0          0          0      35277      35287      19971          0          0          5
  8   12190188      70427          0          0          0          0      35211      35216          0          0          0          0
  9   12182774      70388          0          0          0          0      35191      35197          0          0          0          0
 10   12188355      70379          0          0          0          0      35184      35195          0          0          0          0
 11   12186185      70377          0          0          0          0      35181      35196          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          7          0          0          1          0          0          0          0          0
  2          2          0          0          2          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7         23          0          0         35          0          1          0          0          0
  8          1          0          0          1          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:52824.328s (    0.342 blocks/s,   0.021 Mb/s)

Attachment 2: WR_sample_R3_0_24Jan19
ts: 0x157CCDB0B142680C dt: 0x0000000001154640 tag: 3 type: 0
 ***            ADC data: block:  7222 ptr:  6283 data: 0xC1ED9102 module:  7 fail: 0 range: 0 id:   493 channel: 45 adc: 37122  ts: 0x157CCDB0C3E06D9C OLD: ts: 0x157CCDB0C6318A80 dt: 0x0000000002511CE4 tag: 3 type: 0
 ***            ADC data: block:  7223 ptr: 10735 data: 0xC1C08FF2 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 36850  ts: 0x157CCDB0C54C53FC OLD: ts: 0x157CCDB0CDC7A53C dt: 0x00000000087B5140 tag: 3 type: 0
 ***            ADC data: block:  7231 ptr:  3151 data: 0xC1D491CE module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 37326  ts: 0x157CCDB0F35EE7EC OLD: ts: 0x157CCDB0F35EEE90 dt: 0x00000000000006A4 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7236 ptr:  5557 data: 0x824CDAE2 module:  2 information type:  4 information field: 0x000CDAE2 ts: 0x157CCDAE20051350 OLD: ts: 0x157CCDB100051350 dt: 0x00000002E0000000 tag: 2 type: 5
 ***            ADC data: block:  7244 ptr: 10247 data: 0xC241910C module:  9 fail: 0 range: 0 id:   577 channel:  1 adc: 37132  ts: 0x157CCDB12BD3FC76 OLD: ts: 0x157CCDB12C4B3D76 dt: 0x0000000000774100 tag: 3 type: 0
 ***            ADC data: block:  7244 ptr: 10607 data: 0xC25A91DE module:  9 fail: 0 range: 0 id:   602 channel: 26 adc: 37342  ts: 0x157CCDB11C4B3D76 OLD: ts: 0x157CCDB11E650552 dt: 0x000000000219C7DC tag: 2 type: 3
 ***            ADC data: block:  7262 ptr: 11789 data: 0xC01D6DBE module:  0 fail: 0 range: 0 id:    29 channel: 29 adc: 28094  ts: 0x157CCDB153D32B7C OLD: ts: 0x157CCDB15BF7DD2C dt: 0x000000000824B1B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7262 ptr: 16349 data: 0x804CDAFE module:  0 information type:  4 information field: 0x000CDAFE ts: 0x157CCDAFE6000000 OLD: ts: 0x157CCDB156000000 dt: 0x0000000170000000 tag: 2 type: 5
 ***            ADC data: block:  7263 ptr:  4089 data: 0xC03E6E65 module:  0 fail: 0 range: 0 id:    62 channel: 62 adc: 28261  ts: 0x157CCDB11BF7DD2C OLD: ts: 0x157CCDB11D32AE3C dt: 0x00000000013AD110 tag: 3 type: 0
 ***            ADC data: block:  7273 ptr: 16067 data: 0xC27E91EE module:  9 fail: 0 range: 0 id:   638 channel: 62 adc: 37358  ts: 0x157CCDB17057B786 OLD: ts: 0x157CCDB170CA8C86 dt: 0x000000000072D500 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7274 ptr:  7515 data: 0x894CDB14 module:  9 information type:  4 information field: 0x000CDB14 ts: 0x157CCDB144000000 OLD: ts: 0x157CCDB174000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  7274 ptr: 14043 data: 0xC0046C82 module:  0 fail: 0 range: 0 id:     4 channel:  4 adc: 27778  ts: 0x157CCDB140CA8D6C OLD: ts: 0x157CCDB147068076 dt: 0x00000000063BF30A tag: 3 type: 0
 ***            ADC data: block:  7280 ptr: 11927 data: 0xC25591B6 module:  9 fail: 0 range: 0 id:   597 channel: 21 adc: 37302  ts: 0x157CCDB170CA9456 OLD: ts: 0x157CCDB17E283DA6 dt: 0x000000000D5DA950 tag: 3 type: 0
 ***            ADC data: block:  7288 ptr:  8471 data: 0xC27C9F0C module:  9 fail: 0 range: 0 id:   636 channel: 60 adc: 40716  ts: 0x157CCDB1928102F6 OLD: ts: 0x157CCDB1928103DC dt: 0x00000000000000E6 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7291 ptr:  5241 data: 0x894CDB18 module:  9 information type:  4 information field: 0x000CDB18 ts: 0x157CCDB180000000 OLD: ts: 0x157CCDB190000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7300 ptr:  8843 data: 0xC0A36F23 module:  2 fail: 0 range: 0 id:   163 channel: 35 adc: 28451  ts: 0x157CCDB1B6A96880 OLD: ts: 0x157CCDB1B6A96C7C dt: 0x00000000000003FC tag: 3 type: 0
 ***            ADC data: block:  7307 ptr:  7089 data: 0xC2B56CFC module: 10 fail: 0 range: 0 id:   693 channel: 53 adc: 27900  ts: 0x157CCDB1CDA254B6 OLD: ts: 0x157CCDB1CDA2581C dt: 0x0000000000000366 tag: 3 type: 0
 ***            ADC data: block:  7311 ptr: 14317 data: 0xC1806EB3 module:  6 fail: 0 range: 0 id:   384 channel:  0 adc: 28339  ts: 0x157CCDB1D71DC60E OLD: ts: 0x157CCDB1DC6E789E dt: 0x000000000550B290 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7311 ptr: 16239 data: 0x864CDA7A module:  6 information type:  4 information field: 0x000CDA7A ts: 0x157CCDA7A8000000 OLD: ts: 0x157CCDB1D8000000 dt: 0x0000000A30000000 tag: 2 type: 5
 ***            ADC data: block:  7312 ptr: 12293 data: 0xC1B56D5D module:  6 fail: 0 range: 0 id:   437 channel: 53 adc: 27997  ts: 0x157CCDA7AC6E789E OLD: ts: 0x157CCDA7ADCC96CE dt: 0x00000000015E1E30 tag: 3 type: 0
 ***            ADC data: block:  7324 ptr: 12065 data: 0xC1896DBD module:  6 fail: 0 range: 0 id:   393 channel:  9 adc: 28093  ts: 0x157CCDB1F9CC29BE OLD: ts: 0x157CCDB1FFC1709E dt: 0x0000000005F546E0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7325 ptr:  4711 data: 0x864CDB1E module:  6 information type:  4 information field: 0x000CDB1E ts: 0x157CCDB1EE000000 OLD: ts: 0x157CCDB1FE000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7339 ptr: 13583 data: 0xC23453E4 module:  8 fail: 0 range: 0 id:   564 channel: 52 adc: 21476  ts: 0x157CCDB220A1996A OLD: ts: 0x157CCDB220A19BAE dt: 0x0000000000000244 tag: 3 type: 0
 ***            ADC data: block:  7367 ptr:  9639 data: 0xC28B6B14 module: 10 fail: 0 range: 0 id:   651 channel: 11 adc: 27412  ts: 0x157CCDB26CD3C786 OLD: ts: 0x157CCDB26E3095C0 dt: 0x00000000015CCE3A tag: 3 type: 0
 ***            ADC data: block:  7397 ptr:  3995 data: 0xC0836EC4 module:  2 fail: 0 range: 0 id:   131 channel:  3 adc: 28356  ts: 0x157CCDB2A591BF60 OLD: ts: 0x157CCDB2AE7009B0 dt: 0x0000000008DE4A50 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7404 ptr:  2225 data: 0x834CDB2A module:  3 information type:  4 information field: 0x000CDB2A ts: 0x157CCDB2AC000000 OLD: ts: 0x157CCDB2BC000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7404 ptr:  2767 data: 0xC0846BCA module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 27594  ts: 0x157CCDB2A2EF5910 OLD: ts: 0x157CCDB2AC408800 dt: 0x0000000009512EF0 tag: 3 type: 0
 ***            ADC data: block:  7417 ptr:  9149 data: 0xC2956D66 module: 10 fail: 0 range: 0 id:   661 channel: 21 adc: 28006  ts: 0x157CCDB2CB43EE16 OLD: ts: 0x157CCDB2CE9EEF86 dt: 0x00000000035B0170 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7418 ptr:  2977 data: 0x8A4CDB1B module: 10 information type:  4 information field: 0x000CDB1B ts: 0x157CCDB1B009C326 OLD: ts: 0x157CCDB2C009C326 dt: 0x0000000110000000 tag: 2 type: 5
 ***            ADC data: block:  7419 ptr:  5107 data: 0xC2996BF0 module: 10 fail: 0 range: 0 id:   665 channel: 25 adc: 27632  ts: 0x157CCDB1B183D816 OLD: ts: 0x157CCDB1B8A19F36 dt: 0x00000000071DC720 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7420 ptr:  6257 data: 0x8A4CD742 module: 10 information type:  4 information field: 0x000CD742 ts: 0x157CCD7420056696 OLD: ts: 0x157CCDB1B0056696 dt: 0x0000003D90000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  7421 ptr:  4361 data: 0x8A4CD427 module: 10 information type:  4 information field: 0x000CD427 ts: 0x157CCD427C000000 OLD: ts: 0x157CCD742C000000 dt: 0x00000031B0000000 tag: 2 type: 5
 ***            ADC data: block:  7421 ptr: 15591 data: 0xC28B6B12 module: 10 fail: 0 range: 0 id:   651 channel: 11 adc: 27410  ts: 0x157CCD42813C0276 OLD: ts: 0x157CCD428E9EF59E dt: 0x000000000D62F328 tag: 3 type: 0
 ***            ADC data: block:  7422 ptr: 10585 data: 0xC2BD629A module: 10 fail: 0 range: 0 id:   701 channel: 61 adc: 25242  ts: 0x157CCD4286AA2576 OLD: ts: 0x157CCD428E9EF59E dt: 0x0000000007F4D028 tag: 3 type: 0
 ***            ADC data: block:  7428 ptr: 16281 data: 0xC1826E57 module:  6 fail: 0 range: 0 id:   386 channel:  2 adc: 28247  ts: 0x157CCDB2D91F38EE OLD: ts: 0x157CCDB2DB54C750 dt: 0x0000000002358E62 tag: 3 type: 0
 ***            ADC data: block:  7431 ptr:  3465 data: 0xC1AC7005 module:  6 fail: 0 range: 0 id:   428 channel: 44 adc: 28677  ts: 0x157CCDB2DDC738FE OLD: ts: 0x157CCDB2DF07855E dt: 0x0000000001404C60 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7431 ptr:  8231 data: 0x864CDB21 module:  6 information type:  4 information field: 0x000CDB21 ts: 0x157CCDB2100B1E1E OLD: ts: 0x157CCDB2D00B1E1E dt: 0x00000000C0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  7439 ptr: 16303 data: 0x814CDAF0 module:  1 information type:  4 information field: 0x000CDAF0 ts: 0x157CCDAF0A000000 OLD: ts: 0x157CCDB2FA000000 dt: 0x00000003F0000000 tag: 2 type: 5
 ***            ADC data: block:  7440 ptr:  9539 data: 0xC0CB93CC module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 37836  ts: 0x157CCDAF01B4A6F0 OLD: ts: 0x157CCDAF0E82D2E2 dt: 0x000000000CCE2BF2 tag: 3 type: 0
 ***            ADC data: block:  7443 ptr: 13655 data: 0xC06A9291 module:  1 fail: 0 range: 0 id:   106 channel: 42 adc: 37521  ts: 0x157CCDB2F1B4A132 OLD: ts: 0x157CCDB2FFC91F32 dt: 0x000000000E147E00 tag: 3 type: 0
 ***            ADC data: block:  7448 ptr: 12121 data: 0xC0D2916F module:  3 fail: 0 range: 0 id:   210 channel: 18 adc: 37231  ts: 0x157CCDB30CBBB830 OLD: ts: 0x157CCDB30CBBB952 dt: 0x0000000000000122 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7451 ptr: 13555 data: 0x814CDB30 module:  1 information type:  4 information field: 0x000CDB30 ts: 0x157CCDB3000637D2 OLD: ts: 0x157CCDB3100637D2 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7452 ptr: 11031 data: 0xC0C5913F module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 37183  ts: 0x157CCDB304C9E240 OLD: ts: 0x157CCDB30677E822 dt: 0x0000000001AE05E2 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7457 ptr:   481 data: 0x814CDB31 module:  1 information type:  4 information field: 0x000CDB31 ts: 0x157CCDB31A000000 OLD: ts: 0x157CCDB32A000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7457 ptr:  3609 data: 0xC2C890B4 module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 37044  ts: 0x157CCDB312278848 OLD: ts: 0x157CCDB31B78B512 dt: 0x0000000009512CCA tag: 3 type: 0
 ***            ADC data: block:  7471 ptr:  7617 data: 0xC2E390DB module: 11 fail: 0 range: 0 id:   739 channel: 35 adc: 37083  ts: 0x157CCDB3551E46E8 OLD: ts: 0x157CCDB359D2B8F8 dt: 0x0000000004B47210 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7472 ptr:  4649 data: 0x8B4CDB34 module: 11 information type:  4 information field: 0x000CDB34 ts: 0x157CCDB34C000000 OLD: ts: 0x157CCDB35C000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  7472 ptr:  5593 data: 0xC04992C4 module:  1 fail: 0 range: 0 id:    73 channel:  9 adc: 37572  ts: 0x157CCDB349D2BE52 OLD: ts: 0x157CCDB34C751F48 dt: 0x0000000002A260F6 tag: 3 type: 0
 ***            ADC data: block:  7497 ptr:  9601 data: 0xC2F092AD module: 11 fail: 0 range: 0 id:   752 channel: 48 adc: 37549  ts: 0x157CCDB3B3AE3398 OLD: ts: 0x157CCDB3B3AE3802 dt: 0x000000000000046A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7503 ptr: 15867 data: 0x884CDAE7 module:  8 information type:  4 information field: 0x000CDAE7 ts: 0x157CCDAE7008A4CA OLD: ts: 0x157CCDB3C008A4CA dt: 0x0000000550000000 tag: 2 type: 5
 ***            ADC data: block:  7504 ptr:  6735 data: 0xC2106E93 module:  8 fail: 0 range: 0 id:   528 channel: 16 adc: 28307  ts: 0x157CCDAE71D2599A OLD: ts: 0x157CCDAE737EEF6A dt: 0x0000000001AC95D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7506 ptr:  6071 data: 0x844CDAEB module:  4 information type:  4 information field: 0x000CDAEB ts: 0x157CCDAEB0000000 OLD: ts: 0x157CCDB3C0000000 dt: 0x0000000510000000 tag: 2 type: 5
 ***            ADC data: block:  7512 ptr:  9347 data: 0xC0F59188 module:  3 fail: 0 range: 0 id:   245 channel: 53 adc: 37256  ts: 0x157CCDB3D4DAFB50 OLD: ts: 0x157CCDB3DD566CF2 dt: 0x00000000087B71A2 tag: 3 type: 0
 ***            ADC data: block:  7545 ptr: 12913 data: 0xC0FF93D4 module:  3 fail: 0 range: 0 id:   255 channel: 63 adc: 37844  ts: 0x157CCDB40B647FC0 OLD: ts: 0x157CCDB40B6481D2 dt: 0x0000000000000212 tag: 3 type: 0
 ***            ADC data: block:  7546 ptr:  2095 data: 0xC0E59251 module:  3 fail: 0 range: 0 id:   229 channel: 37 adc: 37457  ts: 0x157CCDB402EF58C0 OLD: ts: 0x157CCDB40BEDEB70 dt: 0x0000000008FE92B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7546 ptr:  8709 data: 0x834CDB2B module:  3 information type:  4 information field: 0x000CDB2B ts: 0x157CCDB2B6000000 OLD: ts: 0x157CCDB406000000 dt: 0x0000000150000000 tag: 2 type: 5
 ***            ADC data: block:  7547 ptr:    71 data: 0xC0ED91CD module:  3 fail: 0 range: 0 id:   237 channel: 45 adc: 37325  ts: 0x157CCDB2B055E1B0 OLD: ts: 0x157CCDB2B9EACAB0 dt: 0x000000000994E900 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7547 ptr: 12125 data: 0x834CDAF4 module:  3 information type:  4 information field: 0x000CDAF4 ts: 0x157CCDAF46000000 OLD: ts: 0x157CCDB2B6000000 dt: 0x0000000370000000 tag: 2 type: 5
 ***            ADC data: block:  7554 ptr: 15129 data: 0xC0116E66 module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 28262  ts: 0x157CCDB419E3D99C OLD: ts: 0x157CCDB419E3DA0A dt: 0x000000000000006E tag: 3 type: 0
 ***            ADC data: block:  7613 ptr: 14657 data: 0xC0DA9275 module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 37493  ts: 0x157CCDB4BBC13340 OLD: ts: 0x157CCDB4BCD78040 dt: 0x0000000001164D00 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7614 ptr:  3065 data: 0x834CDB46 module:  3 information type:  4 information field: 0x000CDB46 ts: 0x157CCDB46E000000 OLD: ts: 0x157CCDB4BE000000 dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block:  7615 ptr: 11571 data: 0xC21F6E69 module:  8 fail: 0 range: 0 id:   543 channel: 31 adc: 28265  ts: 0x157CCDB4BF592E9A OLD: ts: 0x157CCDB4BF5931B0 dt: 0x0000000000000316 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7617 ptr: 12341 data: 0x884CDB3F module:  8 information type:  4 information field: 0x000CDB3F ts: 0x157CCDB3F000F38A OLD: ts: 0x157CCDB4C000F38A dt: 0x00000000D0000000 tag: 2 type: 5
 ***            ADC data: block:  7626 ptr:  2431 data: 0xC1509074 module:  5 fail: 0 range: 0 id:   336 channel: 16 adc: 36980  ts: 0x157CCDB4E4C20C9C OLD: ts: 0x157CCDB4EBA5E11C dt: 0x0000000006E3D480 tag: 3 type: 0
 ***            ADC data: block:  7679 ptr: 15149 data: 0xC25A91F7 module:  9 fail: 0 range: 0 id:   602 channel: 26 adc: 37367  ts: 0x157CCDB561F0FC96 OLD: ts: 0x157CCDB56A74A0A6 dt: 0x000000000883A410 tag: 3 type: 0
 ***            ADC data: block:  7691 ptr:  6121 data: 0xC037682F module:  0 fail: 0 range: 0 id:    55 channel: 55 adc: 26671  ts: 0x157CCDB57992703C OLD: ts: 0x157CCDB579CECADC dt: 0x00000000003C5AA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7691 ptr: 11227 data: 0x804CDB16 module:  0 information type:  4 information field: 0x000CDB16 ts: 0x157CCDB16C000000 OLD: ts: 0x157CCDB57C000000 dt: 0x0000000410000000 tag: 2 type: 5
 ***            ADC data: block:  7756 ptr: 11677 data: 0xC15491D4 module:  5 fail: 0 range: 0 id:   340 channel: 20 adc: 37332  ts: 0x157CCDB61119C18C OLD: ts: 0x157CCDB6168D8EBC dt: 0x000000000573CD30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7757 ptr:  1541 data: 0x854CDB50 module:  5 information type:  4 information field: 0x000CDB50 ts: 0x157CCDB504000000 OLD: ts: 0x157CCDB614000000 dt: 0x0000000110000000 tag: 2 type: 5
 ***            ADC data: block:  7757 ptr:  9653 data: 0xC1569352 module:  5 fail: 0 range: 0 id:   342 channel: 22 adc: 37714  ts: 0x157CCDB5068D8EBC OLD: ts: 0x157CCDB507C88A7C dt: 0x00000000013AFBC0 tag: 3 type: 0
 ***            ADC data: block:  7783 ptr: 14273 data: 0xC21A6F3B module:  8 fail: 0 range: 0 id:   538 channel: 26 adc: 28475  ts: 0x157CCDB64E38F7CA OLD: ts: 0x157CCDB64E38FA0E dt: 0x0000000000000244 tag: 3 type: 0
 ***            ADC data: block:  7796 ptr: 15415 data: 0xC0116E6B module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 28267  ts: 0x157CCDB670EBC98C OLD: ts: 0x157CCDB67DB7526C dt: 0x000000000CCB88E0 tag: 3 type: 0
 ***            ADC data: block:  7798 ptr: 13419 data: 0xC01A6ED0 module:  0 fail: 0 range: 0 id:    26 channel: 26 adc: 28368  ts: 0x157CCDB68210469C OLD: ts: 0x157CCDB68B4300FC dt: 0x000000000932BA60 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7802 ptr: 12389 data: 0x864CDB4E module:  6 information type:  4 information field: 0x000CDB4E ts: 0x157CCDB4E00B4EDE OLD: ts: 0x157CCDB6A00B4EDE dt: 0x00000001C0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  7821 ptr: 11767 data: 0x824CDB28 module:  2 information type:  4 information field: 0x000CDB28 ts: 0x157CCDB28009E600 OLD: ts: 0x157CCDB6F009E600 dt: 0x0000000470000000 tag: 2 type: 5
 ***            ADC data: block:  7822 ptr:   175 data: 0xC0BF6C46 module:  2 fail: 0 range: 0 id:   191 channel: 63 adc: 27718  ts: 0x157CCDB280471880 OLD: ts: 0x157CCDB2823F3C90 dt: 0x0000000001F82410 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7825 ptr:  1267 data: 0x8A4CDB69 module: 10 information type:  4 information field: 0x000CDB69 ts: 0x157CCDB69E000000 OLD: ts: 0x157CCDB6FE000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  7871 ptr: 15335 data: 0xC0C08F98 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 36760  ts: 0x157CCDB77A396800 OLD: ts: 0x157CCDB77B993B70 dt: 0x00000000015FD370 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7872 ptr: 11257 data: 0x834CDB72 module:  3 information type:  4 information field: 0x000CDB72 ts: 0x157CCDB720000000 OLD: ts: 0x157CCDB770000000 dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block:  7874 ptr:  9785 data: 0xC07894FE module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 38142  ts: 0x157CCDB7222785D2 OLD: ts: 0x157CCDB72E582EA2 dt: 0x000000000C30A8D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7874 ptr: 13685 data: 0x814CDB32 module:  1 information type:  4 information field: 0x000CDB32 ts: 0x157CCDB324000000 OLD: ts: 0x157CCDB724000000 dt: 0x0000000400000000 tag: 2 type: 5
 ***            ADC data: block:  7972 ptr: 14245 data: 0xC0906D9C module:  2 fail: 0 range: 0 id:   144 channel: 16 adc: 28060  ts: 0x157CCDB815E5A920 OLD: ts: 0x157CCDB816F905E6 dt: 0x0000000001135CC6 tag: 3 type: 0
 ***            ADC data: block:  7975 ptr: 12307 data: 0xC0AB6B17 module:  2 fail: 0 range: 0 id:   171 channel: 43 adc: 27415  ts: 0x157CCDB818925360 OLD: ts: 0x157CCDB81C57A790 dt: 0x0000000003C55430 tag: 3 type: 0
 ***            ADC data: block:  7976 ptr: 10283 data: 0xC28B6B17 module: 10 fail: 0 range: 0 id:   651 channel: 11 adc: 27415  ts: 0x157CCDB81C57A826 OLD: ts: 0x157CCDB81F411C50 dt: 0x0000000002E9742A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  7994 ptr:  5681 data: 0x884CDB62 module:  8 information type:  4 information field: 0x000CDB62 ts: 0x157CCDB6200A0FBA OLD: ts: 0x157CCDB8500A0FBA dt: 0x0000000230000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  7995 ptr: 11805 data: 0x884CDA53 module:  8 information type:  4 information field: 0x000CDA53 ts: 0x157CCDA5300399BA OLD: ts: 0x157CCDB6200399BA dt: 0x00000010F0000000 tag: 2 type: 5
 ***            ADC data: block:  7995 ptr: 13643 data: 0xC2C0905E module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 36958  ts: 0x157CCDA530C9D958 OLD: ts: 0x157CCDA530D99ECA dt: 0x00000000000FC572 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8003 ptr: 11597 data: 0x8A4CDB70 module: 10 information type:  4 information field: 0x000CDB70 ts: 0x157CCDB70C000000 OLD: ts: 0x157CCDB86C000000 dt: 0x0000000160000000 tag: 2 type: 5
 ***            ADC data: block:  8003 ptr: 13825 data: 0xC2C79071 module: 11 fail: 0 range: 0 id:   711 channel:  7 adc: 36977  ts: 0x157CCDB705F8AB88 OLD: ts: 0x157CCDB70D096EE6 dt: 0x000000000710C35E tag: 3 type: 0
 ***            ADC data: block:  8020 ptr:   669 data: 0xC2F99417 module: 11 fail: 0 range: 0 id:   761 channel: 57 adc: 37911  ts: 0x157CCDB8A2C5B3F8 OLD: ts: 0x157CCDB8A2C5BB8C dt: 0x0000000000000794 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8021 ptr: 15843 data: 0x804CDB69 module:  0 information type:  4 information field: 0x000CDB69 ts: 0x157CCDB69008A72C OLD: ts: 0x157CCDB8A008A72C dt: 0x0000000210000000 tag: 2 type: 5
 ***            ADC data: block:  8032 ptr:  9885 data: 0xC1356E43 module:  4 fail: 0 range: 0 id:   309 channel: 53 adc: 28227  ts: 0x157CCDB8D4DAFC72 OLD: ts: 0x157CCDB8D6DC4102 dt: 0x0000000002014490 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8032 ptr: 15925 data: 0x844CDB3D module:  4 information type:  4 information field: 0x000CDB3D ts: 0x157CCDB3D8000000 OLD: ts: 0x157CCDB8D8000000 dt: 0x0000000500000000 tag: 2 type: 5
 ***            ADC data: block:  8033 ptr:  7861 data: 0xC1226ECD module:  4 fail: 0 range: 0 id:   290 channel: 34 adc: 28365  ts: 0x157CCDB3D6DC4102 OLD: ts: 0x157CCDB3DC50B712 dt: 0x0000000005747610 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8036 ptr: 15963 data: 0x804CDB8B module:  0 information type:  4 information field: 0x000CDB8B ts: 0x157CCDB8BA000000 OLD: ts: 0x157CCDB8EA000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  8037 ptr: 11395 data: 0xC2016D4B module:  8 fail: 0 range: 0 id:   513 channel:  1 adc: 27979  ts: 0x157CCDB8B213E16A OLD: ts: 0x157CCDB8BF975B6C dt: 0x000000000D837A02 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8039 ptr:  9777 data: 0x834CDB78 module:  3 information type:  4 information field: 0x000CDB78 ts: 0x157CCDB78E000000 OLD: ts: 0x157CCDB8BE000000 dt: 0x0000000130000000 tag: 2 type: 5
 ***            ADC data: block:  8039 ptr: 13283 data: 0xC2036F36 module:  8 fail: 0 range: 0 id:   515 channel:  3 adc: 28470  ts: 0x157CCDB78589DDEA OLD: ts: 0x157CCDB78FA59E10 dt: 0x000000000A1BC026 tag: 3 type: 0
 ***            ADC data: block:  8040 ptr: 15849 data: 0xC0E69174 module:  3 fail: 0 range: 0 id:   230 channel: 38 adc: 37236  ts: 0x157CCDB917B4B160 OLD: ts: 0x157CCDB917FC4CF4 dt: 0x0000000000479B94 tag: 2 type: 3
 ***   SYNC100 timestamp: block:  8041 ptr:  4419 data: 0x834CDB45 module:  3 information type:  4 information field: 0x000CDB45 ts: 0x157CCDB45A000000 OLD: ts: 0x157CCDB91A000000 dt: 0x00000004C0000000 tag: 2 type: 5
 ***            ADC data: block:  8045 ptr:    67 data: 0xC2286E1A module:  8 fail: 0 range: 0 id:   552 channel: 40 adc: 28186  ts: 0x157CCDB932088D9A OLD: ts: 0x157CCDB9346B68CA dt: 0x000000000262DB30 tag: 3 type: 0
 ***            ADC data: block:  8045 ptr: 14415 data: 0xC23F6C8C module:  8 fail: 0 range: 0 id:   575 channel: 63 adc: 27788  ts: 0x157CCDB9346B68CA OLD: ts: 0x157CCDB938B7568A dt: 0x00000000044BEDC0 tag: 3 type: 0
 ***            ADC data: block:  8084 ptr: 14511 data: 0xC0EC9225 module:  3 fail: 0 range: 0 id:   236 channel: 44 adc: 37413  ts: 0x157CCDB98A84CDA0 OLD: ts: 0x157CCDB98A84CF1C dt: 0x000000000000017C tag: 3 type: 0
 ***            ADC data: block:  8087 ptr:  9337 data: 0xC152909B module:  5 fail: 0 range: 0 id:   338 channel: 18 adc: 37019  ts: 0x157CCDB98CB9AA3C OLD: ts: 0x157CCDB98F1CFFEA dt: 0x00000000026355AE tag: 3 type: 0
 ***            ADC data: block:  8093 ptr: 15405 data: 0xC2116CFC module:  8 fail: 0 range: 0 id:   529 channel: 17 adc: 27900  ts: 0x157CCDB99B67359A OLD: ts: 0x157CCDB99B6737DE dt: 0x0000000000000244 tag: 3 type: 0
 ***            ADC data: block:  8098 ptr: 11763 data: 0xC1A66DE7 module:  6 fail: 0 range: 0 id:   422 channel: 38 adc: 28135  ts: 0x157CCDB9A8D8C89E OLD: ts: 0x157CCDB9AB18C40E dt: 0x00000000023FFB70 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8099 ptr:  1429 data: 0x864CDB6B module:  6 information type:  4 information field: 0x000CDB6B ts: 0x157CCDB6BC000000 OLD: ts: 0x157CCDB9AC000000 dt: 0x00000002F0000000 tag: 2 type: 5
 ***            ADC data: block:  8109 ptr: 15649 data: 0xC1F796D7 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 38615  ts: 0x157CCDB9D194E2FC OLD: ts: 0x157CCDB9D194E48C dt: 0x0000000000000190 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8112 ptr: 15133 data: 0x874CDB52 module:  7 information type:  4 information field: 0x000CDB52 ts: 0x157CCDB52C000000 OLD: ts: 0x157CCDB9DC000000 dt: 0x00000004B0000000 tag: 2 type: 5
 ***            ADC data: block:  8124 ptr:  8957 data: 0xC151925A module:  5 fail: 0 range: 0 id:   337 channel: 17 adc: 37466  ts: 0x157CCDBA0348962C OLD: ts: 0x157CCDBA03489A1E dt: 0x00000000000003F2 tag: 3 type: 0
 ***            ADC data: block:  8126 ptr:    39 data: 0xC169927F module:  5 fail: 0 range: 0 id:   361 channel: 41 adc: 37503  ts: 0x157CCDBA07927A9C OLD: ts: 0x157CCDBA08D5263C dt: 0x000000000142ABA0 tag: 3 type: 0
 ***            ADC data: block:  8126 ptr: 14387 data: 0xC15D90E5 module:  5 fail: 0 range: 0 id:   349 channel: 29 adc: 37093  ts: 0x157CCDBA01FCD44C OLD: ts: 0x157CCDBA0E41438C dt: 0x000000000C446F40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8127 ptr:  6615 data: 0x854CDB44 module:  5 information type:  4 information field: 0x000CDB44 ts: 0x157CCDB446000000 OLD: ts: 0x157CCDBA06000000 dt: 0x00000005C0000000 tag: 2 type: 5
 ***            ADC data: block:  8130 ptr:  1267 data: 0xC1E9919E module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 37278  ts: 0x157CCDBA10C272CC OLD: ts: 0x157CCDBA192ECD9E dt: 0x00000000086C5AD2 tag: 3 type: 0
 ***            ADC data: block:  8130 ptr: 11767 data: 0xC1AE6B73 module:  6 fail: 0 range: 0 id:   430 channel: 46 adc: 27507  ts: 0x157CCDBA1876752E OLD: ts: 0x157CCDBA1E134E3E dt: 0x00000000059CD910 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8131 ptr:  7185 data: 0x864CDB9B module:  6 information type:  4 information field: 0x000CDB9B ts: 0x157CCDB9BE000000 OLD: ts: 0x157CCDBA1E000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block:  8131 ptr:  9743 data: 0xC19A6C6E module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 27758  ts: 0x157CCDB9B07B024E OLD: ts: 0x157CCDB9BF253E1E dt: 0x000000000EAA3BD0 tag: 3 type: 0
 ***            ADC data: block:  8132 ptr: 11671 data: 0xC1F7964B module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 38475  ts: 0x157CCDB9B0C272CC OLD: ts: 0x157CCDB9BFECC15E dt: 0x000000000F2A4E92 tag: 3 type: 0
 ***            ADC data: block:  8140 ptr:  9747 data: 0xC254903E module:  9 fail: 0 range: 0 id:   596 channel: 20 adc: 36926  ts: 0x157CCDBA4B4D2C26 OLD: ts: 0x157CCDBA4DCEDBAC dt: 0x000000000281AF86 tag: 3 type: 0
 ***            ADC data: block:  8148 ptr: 11217 data: 0xC0206D1E module:  0 fail: 0 range: 0 id:    32 channel: 32 adc: 27934  ts: 0x157CCDBA8213E0FC OLD: ts: 0x157CCDBA8CA2E24C dt: 0x000000000A8F0150 tag: 3 type: 0
 ***            ADC data: block:  8149 ptr:  5907 data: 0xC0396D99 module:  0 fail: 0 range: 0 id:    57 channel: 57 adc: 28057  ts: 0x157CCDBA8D78C81C OLD: ts: 0x157CCDBA8D78CEAC dt: 0x0000000000000690 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8155 ptr:  6197 data: 0x894CDB57 module:  9 information type:  4 information field: 0x000CDB57 ts: 0x157CCDB570058266 OLD: ts: 0x157CCDBAB0058266 dt: 0x0000000540000000 tag: 2 type: 5
 ***            ADC data: block:  8156 ptr:  4131 data: 0xC1428FCE module:  5 fail: 0 range: 0 id:   322 channel:  2 adc: 36814  ts: 0x157CCDB5708E216C OLD: ts: 0x157CCDB576A55736 dt: 0x00000000061735CA tag: 3 type: 0
 ***            ADC data: block:  8158 ptr: 12975 data: 0xC2779841 module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 38977  ts: 0x157CCDBAB08E19F6 OLD: ts: 0x157CCDBABDEBB3A6 dt: 0x000000000D5D99B0 tag: 3 type: 0
 ***            ADC data: block:  8161 ptr: 10489 data: 0xC1408FBF module:  5 fail: 0 range: 0 id:   320 channel:  0 adc: 36799  ts: 0x157CCDBAC4031EFC OLD: ts: 0x157CCDBAC6373C4C dt: 0x0000000002341D50 tag: 3 type: 0
 ***            ADC data: block:  8162 ptr:  6829 data: 0xC24E922B module:  9 fail: 0 range: 0 id:   590 channel: 14 adc: 37419  ts: 0x157CCDBAC6373CA6 OLD: ts: 0x157CCDBAC9EB639C dt: 0x0000000003B426F6 tag: 3 type: 0
 ***            ADC data: block:  8182 ptr:  1553 data: 0xC1966C33 module:  6 fail: 0 range: 0 id:   406 channel: 22 adc: 27699  ts: 0x157CCDBB00269C5E OLD: ts: 0x157CCDBB0082466C dt: 0x00000000005BAA0E tag: 3 type: 0
 ***            ADC data: block:  8185 ptr:  2895 data: 0xC2036F2A module:  8 fail: 0 range: 0 id:   515 channel:  3 adc: 28458  ts: 0x157CCDBB01D7EECA OLD: ts: 0x157CCDBB065B02EA dt: 0x0000000004831420 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8185 ptr: 11833 data: 0x884CDB94 module:  8 information type:  4 information field: 0x000CDB94 ts: 0x157CCDB946000000 OLD: ts: 0x157CCDBB06000000 dt: 0x00000001C0000000 tag: 2 type: 5
 ***            ADC data: block:  8186 ptr:   871 data: 0xC232660E module:  8 fail: 0 range: 0 id:   562 channel: 50 adc: 26126  ts: 0x157CCDB9465B02EA OLD: ts: 0x157CCDB94886AFEA dt: 0x00000000022BAD00 tag: 3 type: 0
 ***            ADC data: block:  8251 ptr:  7925 data: 0xC2AA6D73 module: 10 fail: 0 range: 0 id:   682 channel: 42 adc: 28019  ts: 0x157CCDBBA82BF426 OLD: ts: 0x157CCDBBA82BF6A6 dt: 0x0000000000000280 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8253 ptr: 12169 data: 0x8A4CDBB8 module: 10 information type:  4 information field: 0x000CDBB8 ts: 0x157CCDBB80000000 OLD: ts: 0x157CCDBBA0000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  8258 ptr:  3255 data: 0xC2386AE2 module:  8 fail: 0 range: 0 id:   568 channel: 56 adc: 27362  ts: 0x157CCDBBB7721FFA OLD: ts: 0x157CCDBBB7722522 dt: 0x0000000000000528 tag: 3 type: 0
 ***            ADC data: block:  8267 ptr: 14905 data: 0xC09B6C9B module:  2 fail: 0 range: 0 id:   155 channel: 27 adc: 27803  ts: 0x157CCDBBCF12FF30 OLD: ts: 0x157CCDBBCF130246 dt: 0x0000000000000316 tag: 3 type: 0
 ***            ADC data: block:  8269 ptr:  6847 data: 0xC27C9EF8 module:  9 fail: 0 range: 0 id:   636 channel: 60 adc: 40696  ts: 0x157CCDBBDB8FA986 OLD: ts: 0x157CCDBBDEE02D96 dt: 0x0000000003508410 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8269 ptr:  7771 data: 0x894CDA9E module:  9 information type:  4 information field: 0x000CDA9E ts: 0x157CCDA9EC000000 OLD: ts: 0x157CCDBBDC000000 dt: 0x00000011F0000000 tag: 2 type: 5
 ***            ADC data: block:  8270 ptr:  6419 data: 0xC09E6CD3 module:  2 fail: 0 range: 0 id:   158 channel: 30 adc: 27859  ts: 0x157CCDA9F3452350 OLD: ts: 0x157CCDA9F4BDECB0 dt: 0x000000000178C960 tag: 3 type: 0
 ***            ADC data: block:  8271 ptr:  4395 data: 0xC0866C0C module:  2 fail: 0 range: 0 id:   134 channel:  6 adc: 27660  ts: 0x157CCDA9F4AE9EF0 OLD: ts: 0x157CCDA9F9F3E470 dt: 0x0000000005454580 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8271 ptr:  7045 data: 0x824CD8E7 module:  2 information type:  4 information field: 0x000CD8E7 ts: 0x157CCD8E76000000 OLD: ts: 0x157CCDA9F6000000 dt: 0x0000001B80000000 tag: 2 type: 5
 ***            ADC data: block:  8272 ptr:  2371 data: 0xC0956E56 module:  2 fail: 0 range: 0 id:   149 channel: 21 adc: 28246  ts: 0x157CCD8E74BDECB0 OLD: ts: 0x157CCD8E7B7BDCC0 dt: 0x0000000006BDF010 tag: 3 type: 0
 ***            ADC data: block:  8274 ptr: 10225 data: 0xC1166C22 module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 27682  ts: 0x157CCDBC081D43B2 OLD: ts: 0x157CCDBC081D44A2 dt: 0x00000000000000F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8275 ptr: 11595 data: 0x814CDB78 module:  1 information type:  4 information field: 0x000CDB78 ts: 0x157CCDB78E000000 OLD: ts: 0x157CCDBC0E000000 dt: 0x0000000480000000 tag: 2 type: 5
 ***            ADC data: block:  8279 ptr:   635 data: 0xC0449445 module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 37957  ts: 0x157CCDBC21B3A162 OLD: ts: 0x157CCDBC2DB13D5E dt: 0x000000000BFD9BFC tag: 2 type: 3
 ***            ADC data: block:  8286 ptr:  3665 data: 0xC07CA039 module:  1 fail: 0 range: 0 id:   124 channel: 60 adc: 41017  ts: 0x157CCDBC42EA1982 OLD: ts: 0x157CCDBC4C6ED4C2 dt: 0x000000000984BB40 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8286 ptr: 10241 data: 0x814CDBC1 module:  1 information type:  4 information field: 0x000CDBC1 ts: 0x157CCDBC16000000 OLD: ts: 0x157CCDBC46000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block:  8296 ptr:  4175 data: 0xC0FD9B8A module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 39818  ts: 0x157CCDBC72EFD5E0 OLD: ts: 0x157CCDBC75A96370 dt: 0x0000000002B98D90 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8296 ptr: 14947 data: 0x834CDB92 module:  3 information type:  4 information field: 0x000CDB92 ts: 0x157CCDB928000000 OLD: ts: 0x157CCDBC78000000 dt: 0x0000000350000000 tag: 2 type: 5
 ***            ADC data: block:  8297 ptr:  2151 data: 0xC04591D5 module:  1 fail: 0 range: 0 id:    69 channel:  5 adc: 37333  ts: 0x157CCDB925A96582 OLD: ts: 0x157CCDB9299EAE70 dt: 0x0000000003F548EE tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8309 ptr:  7685 data: 0x824CDBC9 module:  2 information type:  4 information field: 0x000CDBC9 ts: 0x157CCDBC9017D6B0 OLD: ts: 0x157CCDBCA017D6B0 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  8310 ptr:  3109 data: 0xC0B45220 module:  2 fail: 0 range: 0 id:   180 channel: 52 adc: 21024  ts: 0x157CCDBC9414EF40 OLD: ts: 0x157CCDBC95C2EB40 dt: 0x0000000001ADFC00 tag: 3 type: 0
 ***            ADC data: block:  8325 ptr: 11301 data: 0xC05B91D1 module:  1 fail: 0 range: 0 id:    91 channel: 27 adc: 37329  ts: 0x157CCDBCC56FA342 OLD: ts: 0x157CCDBCC56FA5B8 dt: 0x0000000000000276 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8336 ptr: 14789 data: 0x844CDBC0 module:  4 information type:  4 information field: 0x000CDBC0 ts: 0x157CCDBC00000642 OLD: ts: 0x157CCDBCD0000642 dt: 0x00000000D0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  8341 ptr: 10267 data: 0x844CDBCD module:  4 information type:  4 information field: 0x000CDBCD ts: 0x157CCDBCDE000000 OLD: ts: 0x157CCDBCEE000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  8380 ptr:   869 data: 0xC2866B60 module: 10 fail: 0 range: 0 id:   646 channel:  6 adc: 27488  ts: 0x157CCDBD5D0976B6 OLD: ts: 0x157CCDBD5D85D936 dt: 0x00000000007C6280 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8380 ptr:  7219 data: 0x8A4CDB71 module: 10 information type:  4 information field: 0x000CDB71 ts: 0x157CCDB71004F3D6 OLD: ts: 0x157CCDBD5004F3D6 dt: 0x0000000640000000 tag: 2 type: 5
 ***            ADC data: block:  8402 ptr:  3835 data: 0xC037684C module:  0 fail: 0 range: 0 id:    55 channel: 55 adc: 26700  ts: 0x157CCDBD864BCFDC OLD: ts: 0x157CCDBD86A2E7CC dt: 0x00000000005717F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8402 ptr:  7457 data: 0x804CDBB0 module:  0 information type:  4 information field: 0x000CDBB0 ts: 0x157CCDBB08000000 OLD: ts: 0x157CCDBD88000000 dt: 0x0000000280000000 tag: 2 type: 5
 ***            ADC data: block:  8403 ptr:   559 data: 0xC0256E7B module:  0 fail: 0 range: 0 id:    37 channel: 37 adc: 28283  ts: 0x157CCDBB06A2E7CC OLD: ts: 0x157CCDBB0C8057EC dt: 0x0000000005DD7020 tag: 3 type: 0
 ***            ADC data: block:  8450 ptr:  3445 data: 0xC0046C74 module:  0 fail: 0 range: 0 id:     4 channel:  4 adc: 27764  ts: 0x157CCDBDC33F64AC OLD: ts: 0x157CCDBDC9A41F8C dt: 0x000000000664BAE0 tag: 2 type: 3
 ***            ADC data: block:  8490 ptr:  2545 data: 0xC1EC93CC module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 37836  ts: 0x157CCDBE08F2D96C OLD: ts: 0x157CCDBE08F2DAFC dt: 0x0000000000000190 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8491 ptr:  1209 data: 0x874CDBA4 module:  7 information type:  4 information field: 0x000CDBA4 ts: 0x157CCDBA40000000 OLD: ts: 0x157CCDBE00000000 dt: 0x00000003C0000000 tag: 2 type: 5
 ***            ADC data: block:  8543 ptr:  7279 data: 0xC0FB92B1 module:  3 fail: 0 range: 0 id:   251 channel: 59 adc: 37553  ts: 0x157CCDBE5306FD20 OLD: ts: 0x157CCDBE589A3B40 dt: 0x0000000005933E20 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8543 ptr:  9391 data: 0x834CDBC8 module:  3 information type:  4 information field: 0x000CDBC8 ts: 0x157CCDBC84000000 OLD: ts: 0x157CCDBE54000000 dt: 0x00000001D0000000 tag: 2 type: 5
 ***            ADC data: block:  8544 ptr:  5255 data: 0xC10E6F2C module:  4 fail: 0 range: 0 id:   270 channel: 14 adc: 28460  ts: 0x157CCDBC889A3C62 OLD: ts: 0x157CCDBC89B5BE40 dt: 0x00000000011B81DE tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8549 ptr:  6303 data: 0x834CDBE5 module:  3 information type:  4 information field: 0x000CDBE5 ts: 0x157CCDBE5A000000 OLD: ts: 0x157CCDBE6A000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  8550 ptr:  1269 data: 0xC107700B module:  4 fail: 0 range: 0 id:   263 channel:  7 adc: 28683  ts: 0x157CCDBE505BEC52 OLD: ts: 0x157CCDBE5F492370 dt: 0x000000000EED371E tag: 3 type: 0
 ***            ADC data: block:  8571 ptr:  1459 data: 0xC1176F26 module:  4 fail: 0 range: 0 id:   279 channel: 23 adc: 28454  ts: 0x157CCDBE7CD1CFD2 OLD: ts: 0x157CCDBE7EE8B1DC dt: 0x000000000216E20A tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8595 ptr:  8643 data: 0x864CDBAD module:  6 information type:  4 information field: 0x000CDBAD ts: 0x157CCDBAD00002FE OLD: ts: 0x157CCDBEC00002FE dt: 0x00000003F0000000 tag: 2 type: 5
 ***            ADC data: block:  8596 ptr:  6463 data: 0xC1C591EF module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 37359  ts: 0x157CCDBAD13244CC OLD: ts: 0x157CCDBAD69F6A6E dt: 0x00000000056D25A2 tag: 3 type: 0
 ***            ADC data: block:  8600 ptr:  4717 data: 0xC1B45356 module:  6 fail: 0 range: 0 id:   436 channel: 52 adc: 21334  ts: 0x157CCDBEC132427E OLD: ts: 0x157CCDBECE8FD45E dt: 0x000000000D5D91E0 tag: 3 type: 0
 ***            ADC data: block:  8612 ptr:  4503 data: 0xC19B6D23 module:  6 fail: 0 range: 0 id:   411 channel: 27 adc: 27939  ts: 0x157CCDBEEBADDEBE OLD: ts: 0x157CCDBEEBADE10C dt: 0x000000000000024E tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8624 ptr: 11909 data: 0x854CDBDE module:  5 information type:  4 information field: 0x000CDBDE ts: 0x157CCDBDEC000000 OLD: ts: 0x157CCDBF1C000000 dt: 0x0000000130000000 tag: 2 type: 5
 ***            ADC data: block:  8625 ptr:  1281 data: 0xC1CE90B6 module:  7 fail: 0 range: 0 id:   462 channel: 14 adc: 37046  ts: 0x157CCDBDE45F15BC OLD: ts: 0x157CCDBDEEA2C57C dt: 0x000000000A43AFC0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8628 ptr:  2517 data: 0x854CDBF1 module:  5 information type:  4 information field: 0x000CDBF1 ts: 0x157CCDBF18000000 OLD: ts: 0x157CCDBF28000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block:  8628 ptr:  9123 data: 0xC1CD9262 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 37474  ts: 0x157CCDBF11BCBF0C OLD: ts: 0x157CCDBF1B0DD86C dt: 0x0000000009511960 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8629 ptr: 13325 data: 0x874CDBE3 module:  7 information type:  4 information field: 0x000CDBE3 ts: 0x157CCDBE300755DC OLD: ts: 0x157CCDBF200755DC dt: 0x00000000F0000000 tag: 2 type: 5
 ***            ADC data: block:  8632 ptr:  7237 data: 0xC2256DEA module:  8 fail: 0 range: 0 id:   549 channel: 37 adc: 28138  ts: 0x157CCDBF33B8A46A OLD: ts: 0x157CCDBF38C453EA dt: 0x00000000050BAF80 tag: 3 type: 0
 ***            ADC data: block:  8633 ptr:  5213 data: 0xC2236F89 module:  8 fail: 0 range: 0 id:   547 channel: 35 adc: 28553  ts: 0x157CCDBF38C453EA OLD: ts: 0x157CCDBF3A676D5A dt: 0x0000000001A31970 tag: 3 type: 0
 ***            ADC data: block:  8637 ptr:  2263 data: 0xC16B9294 module:  5 fail: 0 range: 0 id:   363 channel: 43 adc: 37524  ts: 0x157CCDBF41BCB8CC OLD: ts: 0x157CCDBF4A8C0C6C dt: 0x0000000008CF53A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8637 ptr:  2813 data: 0x854CDBF2 module:  5 information type:  4 information field: 0x000CDBF2 ts: 0x157CCDBF22000000 OLD: ts: 0x157CCDBF42000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  8638 ptr:   239 data: 0xC166925F module:  5 fail: 0 range: 0 id:   358 channel: 38 adc: 37471  ts: 0x157CCDBF209A345C OLD: ts: 0x157CCDBF286B721C dt: 0x0000000007D13DC0 tag: 3 type: 0
 ***            ADC data: block:  8644 ptr:  6371 data: 0xC27E91CD module:  9 fail: 0 range: 0 id:   638 channel: 62 adc: 37325  ts: 0x157CCDBF703E2A86 OLD: ts: 0x157CCDBF73D64556 dt: 0x0000000003981AD0 tag: 3 type: 0
 ***            ADC data: block:  8645 ptr:  4347 data: 0xC2569462 module:  9 fail: 0 range: 0 id:   598 channel: 22 adc: 37986  ts: 0x157CCDBF73D64556 OLD: ts: 0x157CCDBF76FBC086 dt: 0x0000000003257B30 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8646 ptr: 11965 data: 0x864CDBED module:  6 information type:  4 information field: 0x000CDBED ts: 0x157CCDBED00D73AE OLD: ts: 0x157CCDBF800D73AE dt: 0x00000000B0000000 tag: 2 type: 5
 ***            ADC data: block:  8651 ptr:  4201 data: 0xC1B76758 module:  6 fail: 0 range: 0 id:   439 channel: 55 adc: 26456  ts: 0x157CCDBF9A9C612E OLD: ts: 0x157CCDBF9D3EC77E dt: 0x0000000002A26650 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8655 ptr:  3405 data: 0x864CDBFA module:  6 information type:  4 information field: 0x000CDBFA ts: 0x157CCDBFA00C8D7E OLD: ts: 0x157CCDBFC00C8D7E dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block:  8655 ptr: 11779 data: 0xC1896DC0 module:  6 fail: 0 range: 0 id:   393 channel:  9 adc: 28096  ts: 0x157CCDBFA045167E OLD: ts: 0x157CCDBFA3ED906E dt: 0x0000000003A879F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8664 ptr: 12865 data: 0x884CDBFC module:  8 information type:  4 information field: 0x000CDBFC ts: 0x157CCDBFC007BD1A OLD: ts: 0x157CCDBFE007BD1A dt: 0x0000000020000000 tag: 2 type: 5
 *** ERROR: READ I/O error:       5002
                   blocks:       8673
          ADC data format:   70966828 (  175537.4 Hz)
        Other data format:      21679 (      53.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:        716 (       1.8 Hz)
                              SYNC100:      10480 (      25.9 Hz)
                              WR48-63:      10483 (      25.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:        863 (       2.1 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:         38 (       0.1 Hz)
                              SYNC100:        438 (       1.1 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      404.283 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000            10133.775
  2                0.000            11783.934
  3                0.000            12284.664
  4                0.000             9273.776
  5                0.000            13786.168
  6                0.000            10688.373
  7                0.000             9260.323
  8                0.000            10568.292
  9                0.000            14852.882
 10                0.000            15678.965
 11                0.000             9364.052
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5876038       1804          0          0          0         62        871        871          0          0          0          0
  1    5970261       1789          0          0          0         55        867        867          0          0          0          0
  2    6102952       1835          0          0          0         58        888        889          0          0          0          0
  3    5914868       1793          0          0          0         65        864        864          0          0          0          0
  4    5844600       1967          0          0          0         59        954        954          0          0          0          0
  5    5896012       1752          0          0          0         50        851        851          0          0          0          0
  6    5926325       1798          0          0          0         64        867        867          0          0          0          0
  7    6052339       1867          0          0          0         64        901        902          0          0          0          0
  8    5673982       1731          0          0          0         63        834        834          0          0          0          0
  9    6234152       1867          0          0          0         60        903        904          0          0          0          0
 10    5762398       1743          0          0          0         57        843        843          0          0          0          0
 11    5712901       1733          0          0          0         59        837        837          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0         74          0          2         40          0          0          0          0          0
  1         80          0          2         36          0          0          0          0          0
  2         65          0          1         47          0          0          0          0          0
  3         74          0          6         50          0          0          0          0          0
  4         70          0          4         30          0          0          0          0          0
  5         66          0          5         32          0          0          0          0          0
  6         69          0          3         38          0          0          0          0          0
  7         77          0          3         36          0          0          0          0          0
  8         62          0          1         26          0          0          0          0          0
  9         77          0          6         35          0          0          0          0          0
 10         82          0          4         38          0          0          0          0          0
 11         67          0          1         30          0          0          0          0          0
... 25 more lines ...
Attachment 3: WR_sample_R4_0_24Jan19
[td@localhost COREYUSB]$ ~/analyser/analyser R4_0
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***            ADC data: block:   227 ptr:  4677 data: 0xC01B6E25 module:  0 fail: 0 range: 0 id:    27 channel: 27 adc: 28197  ts: 0x157CCF5D2D557EEA OLD: ts: 0x157CCF5D2D557F26 dt: 0x000000000000003C tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:        382
          ADC data format:    3124984 (  302353.4 Hz)
        Other data format:       1686 (     163.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:        843 (      81.6 Hz)
                              WR48-63:        843 (      81.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          1 (       0.1 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       10.336 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000              103.015
  2                0.000               97.982
  3                0.000               94.626
  4                0.000               97.982
  5                0.000               94.626
  6                0.000              103.015
  7                0.000               94.626
  8                0.000               94.626
  9                0.000               84.560
 10                0.000               92.949
 11                0.000               82.882
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     279487        146          0          0          0          0         73         73          0          0          0          0
  1     276519        152          0          0          0          0         76         76          0          0          0          0
  2     272615        146          0          0          0          0         73         73          0          0          0          0
  3     270124        140          0          0          0          0         70         70          0          0          0          0
  4     266911        144          0          0          0          0         72         72          0          0          0          0
  5     255513        140          0          0          0          0         70         70          0          0          0          0
  6     260077        150          0          0          0          0         75         75          0          0          0          0
  7     245811        140          0          0          0          0         70         70          0          0          0          0
  8     254037        140          0          0          0          0         70         70          0          0          0          0
  9     251421        128          0          0          0          0         64         64          0          0          0          0
 10     247948        136          0          0          0          0         68         68          0          0          0          0
 11     244521        124          0          0          0          0         62         62          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          1          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:53164.434s (    0.007 blocks/s,   0.000 Mb/s)

Attachment 4: WR_sample_R4_0_24Jan19.log
[td@localhost COREYUSB]$ ~/analyser/analyser v R4_0 | more
 *** TDR format 3.3.0 analyser - TD - May 2018
  verbose
 ***   WR48-64 timestamp: block:     1 ptr:     7 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157C00000A000000
 ***   SYNC100 timestamp: block:     1 ptr:     9 data: 0x814CF5B2 module:  1 information type:  4 information field: 0x000CF5B2 ts: 0x157CCF5B2A000000
 ***   WR48-64 timestamp: block:     1 ptr:    11 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B2A000000
 ***   SYNC100 timestamp: block:     1 ptr:    13 data: 0x864CF5B2 module:  6 information type:  4 information field: 0x000CF5B2 ts: 0x157CCF5B2A000000
 ***   WR48-64 timestamp: block:     1 ptr:    15 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B24000000
 ***   SYNC100 timestamp: block:     1 ptr:    17 data: 0x814CF5B3 module:  1 information type:  4 information field: 0x000CF5B3 ts: 0x157CCF5B34000000
 ***   WR48-64 timestamp: block:     1 ptr:    19 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B34000000
 ***   SYNC100 timestamp: block:     1 ptr:    21 data: 0x864CF5B3 module:  6 information type:  4 information field: 0x000CF5B3 ts: 0x157CCF5B34000000
 ***   WR48-64 timestamp: block:     1 ptr:    23 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B3E000000
 ***   SYNC100 timestamp: block:     1 ptr:    25 data: 0x814CF5B3 module:  1 information type:  4 information field: 0x000CF5B3 ts: 0x157CCF5B3E000000
 ***   WR48-64 timestamp: block:     1 ptr:    27 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B3E000000
 ***   SYNC100 timestamp: block:     1 ptr:    29 data: 0x864CF5B3 module:  6 information type:  4 information field: 0x000CF5B3 ts: 0x157CCF5B3E000000
 ***   WR48-64 timestamp: block:     1 ptr:    31 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B38000000
 ***   SYNC100 timestamp: block:     1 ptr:    33 data: 0x804CF5B4 module:  0 information type:  4 information field: 0x000CF5B4 ts: 0x157CCF5B48000000
 ***   WR48-64 timestamp: block:     1 ptr:    35 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B48000000
 ***   SYNC100 timestamp: block:     1 ptr:    37 data: 0x814CF5B4 module:  1 information type:  4 information field: 0x000CF5B4 ts: 0x157CCF5B48000000
 ***   WR48-64 timestamp: block:     1 ptr:    39 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B48000000
 ***   SYNC100 timestamp: block:     1 ptr:    41 data: 0x824CF5B4 module:  2 information type:  4 information field: 0x000CF5B4 ts: 0x157CCF5B48000000
 ***   WR48-64 timestamp: block:     1 ptr:    43 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B48000000
 ***   SYNC100 timestamp: block:     1 ptr:    45 data: 0x844CF5B4 module:  4 information type:  4 information field: 0x000CF5B4 ts: 0x157CCF5B48000000
 ***   WR48-64 timestamp: block:     1 ptr:    47 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B48000000
 ***   SYNC100 timestamp: block:     1 ptr:    49 data: 0x864CF5B4 module:  6 information type:  4 information field: 0x000CF5B4 ts: 0x157CCF5B48000000
 ***   WR48-64 timestamp: block:     1 ptr:    51 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B42000000
 ***   SYNC100 timestamp: block:     1 ptr:    53 data: 0x804CF5B5 module:  0 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B52000000
 ***   WR48-64 timestamp: block:     1 ptr:    55 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B52000000
 ***   SYNC100 timestamp: block:     1 ptr:    57 data: 0x814CF5B5 module:  1 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B52000000
 ***   WR48-64 timestamp: block:     1 ptr:    59 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B52000000
 ***   SYNC100 timestamp: block:     1 ptr:    61 data: 0x824CF5B5 module:  2 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B52000000
 ***   WR48-64 timestamp: block:     1 ptr:    63 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B52000000
 ***   SYNC100 timestamp: block:     1 ptr:    65 data: 0x844CF5B5 module:  4 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B52000000
 ***   WR48-64 timestamp: block:     1 ptr:    67 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B52000000
 ***   SYNC100 timestamp: block:     1 ptr:    69 data: 0x864CF5B5 module:  6 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B52000000
 ***   WR48-64 timestamp: block:     1 ptr:    71 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    73 data: 0x804CF5B5 module:  0 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    75 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    77 data: 0x814CF5B5 module:  1 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    79 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    81 data: 0x824CF5B5 module:  2 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    83 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    85 data: 0x834CF5B5 module:  3 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    87 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    89 data: 0x844CF5B5 module:  4 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    91 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    93 data: 0x854CF5B5 module:  5 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    95 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:    97 data: 0x864CF5B5 module:  6 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:    99 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:   101 data: 0x874CF5B5 module:  7 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:   103 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B5C000000
 ***   SYNC100 timestamp: block:     1 ptr:   105 data: 0x884CF5B5 module:  8 information type:  4 information field: 0x000CF5B5 ts: 0x157CCF5B5C000000
 ***   WR48-64 timestamp: block:     1 ptr:   107 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B56000000
 ***   SYNC100 timestamp: block:     1 ptr:   109 data: 0x804CF5B6 module:  0 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   111 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   113 data: 0x814CF5B6 module:  1 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   115 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   117 data: 0x824CF5B6 module:  2 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   119 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   121 data: 0x834CF5B6 module:  3 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   123 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   125 data: 0x844CF5B6 module:  4 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   127 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   129 data: 0x854CF5B6 module:  5 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   131 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   133 data: 0x864CF5B6 module:  6 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   135 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   137 data: 0x874CF5B6 module:  7 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   139 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   141 data: 0x884CF5B6 module:  8 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   143 data: 0x8A50157C module: 10 information type:  5 information field: 0x0000157C ts: 0x157CCF5B66000000
 ***   SYNC100 timestamp: block:     1 ptr:   145 data: 0x8A4CF5B6 module: 10 information type:  4 information field: 0x000CF5B6 ts: 0x157CCF5B66000000
 ***   WR48-64 timestamp: block:     1 ptr:   147 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B60000000
 ***   SYNC100 timestamp: block:     1 ptr:   149 data: 0x804CF5B7 module:  0 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   151 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   153 data: 0x814CF5B7 module:  1 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   155 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   157 data: 0x824CF5B7 module:  2 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   159 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   161 data: 0x834CF5B7 module:  3 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   163 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   165 data: 0x844CF5B7 module:  4 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   167 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   169 data: 0x854CF5B7 module:  5 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   171 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   173 data: 0x864CF5B7 module:  6 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   175 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   177 data: 0x874CF5B7 module:  7 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   179 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   181 data: 0x884CF5B7 module:  8 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   183 data: 0x8A50157C module: 10 information type:  5 information field: 0x0000157C ts: 0x157CCF5B70000000
 ***   SYNC100 timestamp: block:     1 ptr:   185 data: 0x8A4CF5B7 module: 10 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B70000000
 ***   WR48-64 timestamp: block:     1 ptr:   187 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   189 data: 0x804CF5B7 module:  0 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   191 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   193 data: 0x814CF5B7 module:  1 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   195 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   197 data: 0x824CF5B7 module:  2 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   199 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   201 data: 0x834CF5B7 module:  3 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   203 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   205 data: 0x844CF5B7 module:  4 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   207 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   209 data: 0x854CF5B7 module:  5 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   211 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   213 data: 0x864CF5B7 module:  6 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   215 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   217 data: 0x874CF5B7 module:  7 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   219 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   221 data: 0x884CF5B7 module:  8 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   223 data: 0x8A50157C module: 10 information type:  5 information field: 0x0000157C ts: 0x157CCF5B7A000000
 ***   SYNC100 timestamp: block:     1 ptr:   225 data: 0x8A4CF5B7 module: 10 information type:  4 information field: 0x000CF5B7 ts: 0x157CCF5B7A000000
 ***   WR48-64 timestamp: block:     1 ptr:   227 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B74000000
 ***   SYNC100 timestamp: block:     1 ptr:   229 data: 0x804CF5B8 module:  0 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   231 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   233 data: 0x814CF5B8 module:  1 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   235 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   237 data: 0x824CF5B8 module:  2 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   239 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   241 data: 0x834CF5B8 module:  3 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   243 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   245 data: 0x844CF5B8 module:  4 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   247 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   249 data: 0x854CF5B8 module:  5 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   251 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   253 data: 0x864CF5B8 module:  6 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   255 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   257 data: 0x874CF5B8 module:  7 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   259 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   261 data: 0x884CF5B8 module:  8 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   263 data: 0x8A50157C module: 10 information type:  5 information field: 0x0000157C ts: 0x157CCF5B84000000
 ***   SYNC100 timestamp: block:     1 ptr:   265 data: 0x8A4CF5B8 module: 10 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B84000000
 ***   WR48-64 timestamp: block:     1 ptr:   267 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   269 data: 0x804CF5B8 module:  0 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   271 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   273 data: 0x814CF5B8 module:  1 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   275 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   277 data: 0x824CF5B8 module:  2 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   279 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   281 data: 0x834CF5B8 module:  3 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   283 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   285 data: 0x844CF5B8 module:  4 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   287 data: 0x8550157C module:  5 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   289 data: 0x854CF5B8 module:  5 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   291 data: 0x8650157C module:  6 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   293 data: 0x864CF5B8 module:  6 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   295 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   297 data: 0x874CF5B8 module:  7 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   299 data: 0x8850157C module:  8 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   301 data: 0x884CF5B8 module:  8 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   303 data: 0x8A50157C module: 10 information type:  5 information field: 0x0000157C ts: 0x157CCF5B8E000000
 ***   SYNC100 timestamp: block:     1 ptr:   305 data: 0x8A4CF5B8 module: 10 information type:  4 information field: 0x000CF5B8 ts: 0x157CCF5B8E000000
 ***   WR48-64 timestamp: block:     1 ptr:   307 data: 0x8050157C module:  0 information type:  5 information field: 0x0000157C ts: 0x157CCF5B88000000
 ***   SYNC100 timestamp: block:     1 ptr:   309 data: 0x804CF5B9 module:  0 information type:  4 information field: 0x000CF5B9 ts: 0x157CCF5B98000000
 ***   WR48-64 timestamp: block:     1 ptr:   311 data: 0x8150157C module:  1 information type:  5 information field: 0x0000157C ts: 0x157CCF5B98000000
 ***   SYNC100 timestamp: block:     1 ptr:   313 data: 0x814CF5B9 module:  1 information type:  4 information field: 0x000CF5B9 ts: 0x157CCF5B98000000
 ***   WR48-64 timestamp: block:     1 ptr:   315 data: 0x8250157C module:  2 information type:  5 information field: 0x0000157C ts: 0x157CCF5B98000000
 ***   SYNC100 timestamp: block:     1 ptr:   317 data: 0x824CF5B9 module:  2 information type:  4 information field: 0x000CF5B9 ts: 0x157CCF5B98000000
 ***   WR48-64 timestamp: block:     1 ptr:   319 data: 0x8350157C module:  3 information type:  5 information field: 0x0000157C ts: 0x157CCF5B98000000
 ***   SYNC100 timestamp: block:     1 ptr:   321 data: 0x834CF5B9 module:  3 information type:  4 information field: 0x000CF5B9 ts: 0x157CCF5B98000000
 ***   WR48-64 timestamp: block:     1 ptr:   323 data: 0x8450157C module:  4 information type:  5 information field: 0x0000157C ts: 0x157CCF5B98000000
 ***   SYNC100 timestamp: block:     1 ptr:   325 data: 0x844CF5B9 module:  4 information type:  4 information field: 0x000CF5B9 ts: 0x157CCF5B98000000
/ADC
...skipping
 ***            ADC data: block:     1 ptr:  1503 data: 0xC0006ED9 module:  0 fail: 0 range: 0 id:     0 channel:  0 adc: 28377  ts: 0x157CCF5C89F8B1EA
 ***            ADC data: block:     1 ptr:  1505 data: 0xC0106E4D module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 28237  ts: 0x157CCF5C89F8B1EA
 ***            ADC data: block:     1 ptr:  1507 data: 0xC0206D0F module:  0 fail: 0 range: 0 id:    32 channel: 32 adc: 27919  ts: 0x157CCF5C89F8B1EA
 ***            ADC data: block:     1 ptr:  1509 data: 0xC0306F07 module:  0 fail: 0 range: 0 id:    48 channel: 48 adc: 28423  ts: 0x157CCF5C89F8B1EA
 ***            ADC data: block:     1 ptr:  1511 data: 0xC0016E54 module:  0 fail: 0 range: 0 id:     1 channel:  1 adc: 28244  ts: 0x157CCF5C89F8B9BA
 ***            ADC data: block:     1 ptr:  1513 data: 0xC0116E5E module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 28254  ts: 0x157CCF5C89F8B9BA
 ***            ADC data: block:     1 ptr:  1515 data: 0xC0216B59 module:  0 fail: 0 range: 0 id:    33 channel: 33 adc: 27481  ts: 0x157CCF5C89F8B9BA
 ***            ADC data: block:     1 ptr:  1517 data: 0xC0316DE8 module:  0 fail: 0 range: 0 id:    49 channel: 49 adc: 28136  ts: 0x157CCF5C89F8B9BA
 ***            ADC data: block:     1 ptr:  1519 data: 0xC0026E86 module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 28294  ts: 0x157CCF5C89F8C18A
 ***            ADC data: block:     1 ptr:  1521 data: 0xC0126E54 module:  0 fail: 0 range: 0 id:    18 channel: 18 adc: 28244  ts: 0x157CCF5C89F8C18A
 ***            ADC data: block:     1 ptr:  1523 data: 0xC0226E3C module:  0 fail: 0 range: 0 id:    34 channel: 34 adc: 28220  ts: 0x157CCF5C89F8C18A
 ***            ADC data: block:     1 ptr:  1525 data: 0xC0326698 module:  0 fail: 0 range: 0 id:    50 channel: 50 adc: 26264  ts: 0x157CCF5C89F8C18A
 ***            ADC data: block:     1 ptr:  1527 data: 0xC0037006 module:  0 fail: 0 range: 0 id:     3 channel:  3 adc: 28678  ts: 0x157CCF5C89F8C95A
 ***            ADC data: block:     1 ptr:  1529 data: 0xC0136E3C module:  0 fail: 0 range: 0 id:    19 channel: 19 adc: 28220  ts: 0x157CCF5C89F8C95A
 ***            ADC data: block:     1 ptr:  1531 data: 0xC0236FB9 module:  0 fail: 0 range: 0 id:    35 channel: 35 adc: 28601  ts: 0x157CCF5C89F8C95A
 ***            ADC data: block:     1 ptr:  1533 data: 0xC0336B63 module:  0 fail: 0 range: 0 id:    51 channel: 51 adc: 27491  ts: 0x157CCF5C89F8C95A
 ***            ADC data: block:     1 ptr:  1535 data: 0xC0046C5C module:  0 fail: 0 range: 0 id:     4 channel:  4 adc: 27740  ts: 0x157CCF5C89F8D12A
 ***            ADC data: block:     1 ptr:  1537 data: 0xC0146E1E module:  0 fail: 0 range: 0 id:    20 channel: 20 adc: 28190  ts: 0x157CCF5C89F8D12A
 ***            ADC data: block:     1 ptr:  1539 data: 0xC0246C67 module:  0 fail: 0 range: 0 id:    36 channel: 36 adc: 27751  ts: 0x157CCF5C89F8D12A
 ***            ADC data: block:     1 ptr:  1541 data: 0xC03454D2 module:  0 fail: 0 range: 0 id:    52 channel: 52 adc: 21714  ts: 0x157CCF5C89F8D12A
 ***            ADC data: block:     1 ptr:  1543 data: 0xC0056E7F module:  0 fail: 0 range: 0 id:     5 channel:  5 adc: 28287  ts: 0x157CCF5C89F8D8FA
 ***            ADC data: block:     1 ptr:  1545 data: 0xC0156E67 module:  0 fail: 0 range: 0 id:    21 channel: 21 adc: 28263  ts: 0x157CCF5C89F8D8FA
 ***            ADC data: block:     1 ptr:  1547 data: 0xC0256E66 module:  0 fail: 0 range: 0 id:    37 channel: 37 adc: 28262  ts: 0x157CCF5C89F8D8FA
 ***            ADC data: block:     1 ptr:  1549 data: 0xC0356D20 module:  0 fail: 0 range: 0 id:    53 channel: 53 adc: 27936  ts: 0x157CCF5C89F8D8FA
 ***            ADC data: block:     1 ptr:  1551 data: 0xC0066B7D module:  0 fail: 0 range: 0 id:     6 channel:  6 adc: 27517  ts: 0x157CCF5C89F8E0CA
 ***            ADC data: block:     1 ptr:  1553 data: 0xC0166D23 module:  0 fail: 0 range: 0 id:    22 channel: 22 adc: 27939  ts: 0x157CCF5C89F8E0CA
 ***            ADC data: block:     1 ptr:  1555 data: 0xC0266E6A module:  0 fail: 0 range: 0 id:    38 channel: 38 adc: 28266  ts: 0x157CCF5C89F8E0CA
 ***            ADC data: block:     1 ptr:  1557 data: 0xC0366E32 module:  0 fail: 0 range: 0 id:    54 channel: 54 adc: 28210  ts: 0x157CCF5C89F8E0CA
 ***            ADC data: block:     1 ptr:  1559 data: 0xC0076E75 module:  0 fail: 0 range: 0 id:     7 channel:  7 adc: 28277  ts: 0x157CCF5C89F8E89A
 ***            ADC data: block:     1 ptr:  1561 data: 0xC0176E4A module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 28234  ts: 0x157CCF5C89F8E89A
 ***            ADC data: block:     1 ptr:  1563 data: 0xC0276D76 module:  0 fail: 0 range: 0 id:    39 channel: 39 adc: 28022  ts: 0x157CCF5C89F8E89A
 ***            ADC data: block:     1 ptr:  1565 data: 0xC037681D module:  0 fail: 0 range: 0 id:    55 channel: 55 adc: 26653  ts: 0x157CCF5C89F8E89A
 ***            ADC data: block:     1 ptr:  1567 data: 0xC0086FBB module:  0 fail: 0 range: 0 id:     8 channel:  8 adc: 28603  ts: 0x157CCF5C89F8F06A
 ***            ADC data: block:     1 ptr:  1569 data: 0xC0186CBF module:  0 fail: 0 range: 0 id:    24 channel: 24 adc: 27839  ts: 0x157CCF5C89F8F06A
 ***            ADC data: block:     1 ptr:  1571 data: 0xC0286D87 module:  0 fail: 0 range: 0 id:    40 channel: 40 adc: 28039  ts: 0x157CCF5C89F8F06A
 ***            ADC data: block:     1 ptr:  1573 data: 0xC0386B15 module:  0 fail: 0 range: 0 id:    56 channel: 56 adc: 27413  ts: 0x157CCF5C89F8F06A
 ***            ADC data: block:     1 ptr:  1575 data: 0xC0096E7F module:  0 fail: 0 range: 0 id:     9 channel:  9 adc: 28287  ts: 0x157CCF5C89F8F83A
 ***            ADC data: block:     1 ptr:  1577 data: 0xC0196CFF module:  0 fail: 0 range: 0 id:    25 channel: 25 adc: 27903  ts: 0x157CCF5C89F8F83A
 ***            ADC data: block:     1 ptr:  1579 data: 0xC0296DB8 module:  0 fail: 0 range: 0 id:    41 channel: 41 adc: 28088  ts: 0x157CCF5C89F8F83A
 ***            ADC data: block:     1 ptr:  1581 data: 0xC0396DA0 module:  0 fail: 0 range: 0 id:    57 channel: 57 adc: 28064  ts: 0x157CCF5C89F8F83A
 ***            ADC data: block:     1 ptr:  1583 data: 0xC00A6EED module:  0 fail: 0 range: 0 id:    10 channel: 10 adc: 28397  ts: 0x157CCF5C89F9000A
 ***            ADC data: block:     1 ptr:  1585 data: 0xC01A6ECA module:  0 fail: 0 range: 0 id:    26 channel: 26 adc: 28362  ts: 0x157CCF5C89F9000A
 ***            ADC data: block:     1 ptr:  1587 data: 0xC02A6DA3 module:  0 fail: 0 range: 0 id:    42 channel: 42 adc: 28067  ts: 0x157CCF5C89F9000A
 ***            ADC data: block:     1 ptr:  1589 data: 0xC03A6C96 module:  0 fail: 0 range: 0 id:    58 channel: 58 adc: 27798  ts: 0x157CCF5C89F9000A
 ***            ADC data: block:     1 ptr:  1591 data: 0xC00B6DDB module:  0 fail: 0 range: 0 id:    11 channel: 11 adc: 28123  ts: 0x157CCF5C89F907DA
 ***            ADC data: block:     1 ptr:  1593 data: 0xC01B6E0C module:  0 fail: 0 range: 0 id:    27 channel: 27 adc: 28172  ts: 0x157CCF5C89F907DA
 ***            ADC data: block:     1 ptr:  1595 data: 0xC02B6C96 module:  0 fail: 0 range: 0 id:    43 channel: 43 adc: 27798  ts: 0x157CCF5C89F907DA
 ***            ADC data: block:     1 ptr:  1597 data: 0xC03B6D0C module:  0 fail: 0 range: 0 id:    59 channel: 59 adc: 27916  ts: 0x157CCF5C89F907DA
 ***            ADC data: block:     1 ptr:  1599 data: 0xC00C6E91 module:  0 fail: 0 range: 0 id:    12 channel: 12 adc: 28305  ts: 0x157CCF5C89F90FAA
 ***            ADC data: block:     1 ptr:  1601 data: 0xC01C6EBA module:  0 fail: 0 range: 0 id:    28 channel: 28 adc: 28346  ts: 0x157CCF5C89F90FAA
 ***            ADC data: block:     1 ptr:  1603 data: 0xC02C6D0C module:  0 fail: 0 range: 0 id:    44 channel: 44 adc: 27916  ts: 0x157CCF5C89F90FAA
 ***            ADC data: block:     1 ptr:  1605 data: 0xC03C5EDC module:  0 fail: 0 range: 0 id:    60 channel: 60 adc: 24284  ts: 0x157CCF5C89F90FAA
 ***            ADC data: block:     1 ptr:  1607 data: 0xC00D6F63 module:  0 fail: 0 range: 0 id:    13 channel: 13 adc: 28515  ts: 0x157CCF5C89F9177A
 ***            ADC data: block:     1 ptr:  1609 data: 0xC01D6DC6 module:  0 fail: 0 range: 0 id:    29 channel: 29 adc: 28102  ts: 0x157CCF5C89F9177A

  20   Thu Jan 24 13:29:40 2019 TDAnalysis of GSIJan19/R1_0
Analysis of file GSIJan19/R1_0 (white rabbit data)

attachment 1 - overall statistics

attachment 2 - verbose output listing all data around first six identified timewarps

attachment 3 - analyser.f (old version for 48 bit timestamp)

attachment 4 - analyser.f (new version for 64 bit WR timestamp)

Observations

1) File does not commence with information type 4 & 5 data items

   DAQ/TapeServer start sequence issue??

2) DAQ RESUMEs observed for all FEE64s - DAQ PAUSEs *not* observed

3) Number of information type 4 & 5 data items for each FEE64 *not* the same

4) reported idle & elapsed times incorrect - presumably analyser.f bug

 
Attachment 1: WR_sample_R1_0_24jan19
ts: 0x157CBFFD6C000000 dt: 0x00000000C0000000 tag: 2 type: 5
 ***            ADC data: block: 27491 ptr:  5999 data: 0xC1F88255 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33365  ts: 0x157CBFFCA35DCE5E OLD: ts: 0x157CBFFCAC7B99FE dt: 0x00000000091DCBA0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27491 ptr: 14383 data: 0x874BFFBC module:  7 information type:  4 information field: 0x000BFFBC ts: 0x157CBFFBC6000000 OLD: ts: 0x157CBFFCA6000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***            ADC data: block: 27539 ptr:  5675 data: 0xC1EF8149 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33097  ts: 0x157CBFFE42590F6E OLD: ts: 0x157CBFFE4302D526 dt: 0x0000000000A9C5B8 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 27540 ptr: 15597 data: 0x874BFFDA module:  7 information type:  4 information field: 0x000BFFDA ts: 0x157CBFFDA000160E OLD: ts: 0x157CBFFE4000160E dt: 0x00000000A0000000 tag: 2 type: 5
 ***            ADC data: block: 27543 ptr:   149 data: 0xC1CD8134 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33076  ts: 0x157CBFFDA5E4031E OLD: ts: 0x157CBFFDA9CD2E2E dt: 0x0000000003E92B10 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27567 ptr:   183 data: 0x874BFFE8 module:  7 information type:  4 information field: 0x000BFFE8 ts: 0x157CBFFE8000918E OLD: ts: 0x157CBFFEC000918E dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block: 27568 ptr:  4223 data: 0xC1F88165 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33125  ts: 0x157CBFFE82CEB5FE OLD: ts: 0x157CBFFE865EF85E dt: 0x0000000003904260 tag: 3 type: 0
 ***            ADC data: block: 27595 ptr: 12199 data: 0xC1D5801E module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32798  ts: 0x157CBFFF45303E6E OLD: ts: 0x157CBFFF462AD086 dt: 0x0000000000FA9218 tag: 2 type: 6
 ***            ADC data: block: 27620 ptr:  7957 data: 0xC1F881D8 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33240  ts: 0x157CBFFFAA0FD81E OLD: ts: 0x157CBFFFACA78C6E dt: 0x000000000297B450 tag: 3 type: 0
 ***            ADC data: block: 27621 ptr:  5933 data: 0xC1E180C7 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32967  ts: 0x157CBFFFACA7CAEE OLD: ts: 0x157CBFFFAE77C61E dt: 0x0000000001CFFB30 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27627 ptr: 15645 data: 0x874BFFEF module:  7 information type:  4 information field: 0x000BFFEF ts: 0x157CBFFEF0000000 OLD: ts: 0x157CBFFFE0000000 dt: 0x00000000F0000000 tag: 2 type: 5
 ***            ADC data: block: 27630 ptr:  4405 data: 0xC1F981E9 module:  7 fail: 0 range: 0 id:   505 channel: 57 adc: 33257  ts: 0x157CBFFEFA97414E OLD: ts: 0x157CBFFEFDE05C02 dt: 0x0000000003491AB4 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 27630 ptr: 15927 data: 0x874BFFA5 module:  7 information type:  4 information field: 0x000BFFA5 ts: 0x157CBFFA5E000000 OLD: ts: 0x157CBFFEFE000000 dt: 0x00000004A0000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 27631 ptr:  2381 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157CBFFA5B32D92A OLD: ts: 0x157CBFFA5EC867DE dt: 0x0000000003958EB4 tag: 3 type: 0
 ***            ADC data: block: 27667 ptr: 14197 data: 0xC1C580F3 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33011  ts: 0x157CC000ACFF8A9E OLD: ts: 0x157CC000AD90D2CE dt: 0x0000000000914830 tag: 2 type: 6
 ***            ADC data: block: 27668 ptr:  9055 data: 0xC1F28174 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33140  ts: 0x157CC000A9D8DEDE OLD: ts: 0x157CC000AF7A75AE dt: 0x0000000005A196D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27669 ptr:  6321 data: 0x874BFFFF module:  7 information type:  4 information field: 0x000BFFFF ts: 0x157CBFFFFE000000 OLD: ts: 0x157CC000AE000000 dt: 0x00000000B0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 27672 ptr:  7889 data: 0x874C0000 module:  7 information type:  4 information field: 0x000C0000 ts: 0x157CC0000000CC5E OLD: ts: 0x157CC000C000CC5E dt: 0x00000000C0000000 tag: 2 type: 5
 ***            ADC data: block: 27673 ptr:   313 data: 0xC1EF8260 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33376  ts: 0x157CC000027F39DE OLD: ts: 0x157CC0000D5261FE dt: 0x000000000AD32820 tag: 3 type: 0
 ***            ADC data: block: 27675 ptr: 10617 data: 0xC1D57FF5 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32757  ts: 0x157CC0000A1795DE OLD: ts: 0x157CC0000EEF5E7E dt: 0x0000000004D7C8A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27676 ptr:  6567 data: 0x874BFF4B module:  7 information type:  4 information field: 0x000BFF4B ts: 0x157CBFF4BE000000 OLD: ts: 0x157CC0000E000000 dt: 0x0000000B50000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 27676 ptr:  8593 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157CBFF4B9AF7DDE OLD: ts: 0x157CBFF4BEA8A052 dt: 0x0000000004F92274 tag: 2 type: 6
 ***            ADC data: block: 27688 ptr:  4631 data: 0xC1F78173 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33139  ts: 0x157CC0010AD0D07E OLD: ts: 0x157CC0010BB1E906 dt: 0x0000000000E11888 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 27689 ptr:  6073 data: 0x874C000F module:  7 information type:  4 information field: 0x000C000F ts: 0x157CC000F000117E OLD: ts: 0x157CC0010000117E dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block: 27690 ptr:  3259 data: 0xC1EF81C3 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33219  ts: 0x157CC000F17A706E OLD: ts: 0x157CC000F43EF3AE dt: 0x0000000002C48340 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27691 ptr: 15271 data: 0x874C000D module:  7 information type:  4 information field: 0x000C000D ts: 0x157CC000DA000000 OLD: ts: 0x157CC000FA000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block: 27738 ptr:  5325 data: 0xC1D2810D module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 33037  ts: 0x157CC001F82008FE OLD: ts: 0x157CC001F8D86DB8 dt: 0x0000000000B864BA tag: 2 type: 6
 *** FEE64 discriminator: block: 27739 ptr:   959 data: 0x87640100 module:  7 information type:  6 information field: 0x00040100 ts: 0x157CC001F3D2219C OLD: ts: 0x157CC001FB029F1E dt: 0x0000000007307D82 tag: 3 type: 0
 ***            ADC data: block: 27762 ptr: 14597 data: 0xC1C38145 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33093  ts: 0x157CC0026BD9F35E OLD: ts: 0x157CC0026CFB1754 dt: 0x00000000012123F6 tag: 2 type: 6
 ***            ADC data: block: 27769 ptr: 15995 data: 0xC1D580D9 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32985  ts: 0x157CC0028C59E59E OLD: ts: 0x157CC0028CC3329C dt: 0x0000000000694CFE tag: 2 type: 6
 ***            ADC data: block: 27770 ptr: 11189 data: 0xC1F38097 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32919  ts: 0x157CC0028A7B0E8E OLD: ts: 0x157CC0028ED4828E dt: 0x0000000004597400 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27771 ptr:  7075 data: 0x874C0022 module:  7 information type:  4 information field: 0x000C0022 ts: 0x157CC0022E000000 OLD: ts: 0x157CC0028E000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block: 27794 ptr:  1621 data: 0xC1C581EE module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33262  ts: 0x157CC002F74FD30E OLD: ts: 0x157CC002F855DA5A dt: 0x000000000106074C tag: 2 type: 6
 ***            ADC data: block: 27823 ptr: 11389 data: 0xC1EC8220 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33312  ts: 0x157CC0037E60455E OLD: ts: 0x157CC0037ECC9128 dt: 0x00000000006C4BCA tag: 2 type: 6
 ***            ADC data: block: 27824 ptr:   849 data: 0xC1EF813F module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33087  ts: 0x157CC00374DB01CE OLD: ts: 0x157CC0037FC357CE dt: 0x000000000AE85600 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27850 ptr:  5379 data: 0x874C003B module:  7 information type:  4 information field: 0x000C003B ts: 0x157CC003B000D48E OLD: ts: 0x157CC003F000D48E dt: 0x0000000040000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 27851 ptr:  9387 data: 0x874C0015 module:  7 information type:  4 information field: 0x000C0015 ts: 0x157CC0015C000000 OLD: ts: 0x157CC003BC000000 dt: 0x0000000260000000 tag: 2 type: 5
 ***            ADC data: block: 27903 ptr:  7217 data: 0xC1EF824A module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33354  ts: 0x157CC004E80E403E OLD: ts: 0x157CC004ED7AE89E dt: 0x00000000056CA860 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27905 ptr:  1269 data: 0x874C0042 module:  7 information type:  4 information field: 0x000C0042 ts: 0x157CC004200000DE OLD: ts: 0x157CC004E00000DE dt: 0x00000000C0000000 tag: 2 type: 5
 ***            ADC data: block: 27906 ptr: 15501 data: 0xC1EC8256 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33366  ts: 0x157CC0042D7AE89E OLD: ts: 0x157CC0042E7807FE dt: 0x0000000000FD1F60 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 27935 ptr: 16273 data: 0x874C0051 module:  7 information type:  4 information field: 0x000C0051 ts: 0x157CC0051000017E OLD: ts: 0x157CC0058000017E dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 27938 ptr: 14175 data: 0x874C000E module:  7 information type:  4 information field: 0x000C000E ts: 0x157CC000E0003E3E OLD: ts: 0x157CC00510003E3E dt: 0x0000000430000000 tag: 2 type: 5
 ***            ADC data: block: 27939 ptr:  7031 data: 0xC1F881FB module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33275  ts: 0x157CC000E2C885FE OLD: ts: 0x157CC000E3B9421E dt: 0x0000000000F0BC20 tag: 3 type: 0
 ***            ADC data: block: 27945 ptr: 15891 data: 0xC1FAA323 module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 41763  ts: 0x157CC005B86F8B0E OLD: ts: 0x157CC005B86F8C3A dt: 0x000000000000012C tag: 3 type: 0
 ***            ADC data: block: 27946 ptr: 12517 data: 0xC1F181BF module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33215  ts: 0x157CC005B813553E OLD: ts: 0x157CC005BB513E9E dt: 0x00000000033DE960 tag: 3 type: 0
 ***            ADC data: block: 27947 ptr: 10493 data: 0xC1F181AB module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33195  ts: 0x157CC005BB51466E OLD: ts: 0x157CC005BC4E014E dt: 0x0000000000FCBAE0 tag: 3 type: 0
 ***            ADC data: block: 27977 ptr:  4217 data: 0xC1FD8265 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33381  ts: 0x157CC0064A1DB0EE OLD: ts: 0x157CC0064EF8C64E dt: 0x0000000004DB1560 tag: 3 type: 0
 ***            ADC data: block: 27996 ptr: 13449 data: 0xC1F78247 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33351  ts: 0x157CC006AB13634E OLD: ts: 0x157CC006ABB105CC dt: 0x00000000009DA27E tag: 2 type: 6
 ***            ADC data: block: 28029 ptr:  8057 data: 0xC1F38051 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32849  ts: 0x157CC0073D9268BE OLD: ts: 0x157CC0073E620DD0 dt: 0x0000000000CFA512 tag: 2 type: 6
 ***            ADC data: block: 28030 ptr:  2029 data: 0xC1E68087 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32903  ts: 0x157CC00736475DBE OLD: ts: 0x157CC0073FC4D91E dt: 0x00000000097D7B60 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28030 ptr: 13617 data: 0x874C0067 module:  7 information type:  4 information field: 0x000C0067 ts: 0x157CC0067A000000 OLD: ts: 0x157CC0073A000000 dt: 0x00000000C0000000 tag: 2 type: 5
 ***            ADC data: block: 28049 ptr:  6379 data: 0xC1C0825F module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33375  ts: 0x157CC007965DDC2E OLD: ts: 0x157CC0079968C8FE dt: 0x00000000030AECD0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28050 ptr:  2235 data: 0x874C0076 module:  7 information type:  4 information field: 0x000C0076 ts: 0x157CC0076A000000 OLD: ts: 0x157CC0079A000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28050 ptr: 10087 data: 0x874C0059 module:  7 information type:  4 information field: 0x000C0059 ts: 0x157CC0059E000000 OLD: ts: 0x157CC0076E000000 dt: 0x00000001D0000000 tag: 2 type: 5
 ***            ADC data: block: 28080 ptr:  5641 data: 0xC1CD80C1 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 32961  ts: 0x157CC0082F29A35E OLD: ts: 0x157CC0082F30BDC4 dt: 0x0000000000071A66 tag: 2 type: 6
 ***            ADC data: block: 28082 ptr:  8365 data: 0xC1F2814B module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33099  ts: 0x157CC008309FFE1E OLD: ts: 0x157CC0083D4A5B0A dt: 0x000000000CAA5CEC tag: 2 type: 6
 *** FEE64 discriminator: block: 28083 ptr:  6341 data: 0x87638000 module:  7 information type:  6 information field: 0x00038000 ts: 0x157CC00832CABF84 OLD: ts: 0x157CC0083506DAAE dt: 0x00000000023C1B2A tag: 3 type: 0
 ***            ADC data: block: 28085 ptr:  5393 data: 0xC1F4821B module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33307  ts: 0x157CC008450701BE OLD: ts: 0x157CC008480D5A96 dt: 0x00000000030658D8 tag: 2 type: 3
 ***   SYNC100 timestamp: block: 28085 ptr: 14321 data: 0x874C005A module:  7 information type:  4 information field: 0x000C005A ts: 0x157CC005A8000000 OLD: ts: 0x157CC00848000000 dt: 0x00000002A0000000 tag: 2 type: 5
 ***            ADC data: block: 28086 ptr:  3371 data: 0xC1E38237 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33335  ts: 0x157CC005A97FF71E OLD: ts: 0x157CC005AF68085E dt: 0x0000000005E81140 tag: 3 type: 0
 *** FEE64 discriminator: block: 28087 ptr:  1347 data: 0x84640010 module:  4 information type:  6 information field: 0x00040010 ts: 0x157CC005A8DD133A OLD: ts: 0x157CC005ADC6B11E dt: 0x0000000004E99DE4 tag: 3 type: 0
 ***            ADC data: block: 28092 ptr: 14127 data: 0xC1F98305 module:  7 fail: 0 range: 0 id:   505 channel: 57 adc: 33541  ts: 0x157CC0086105DE1E OLD: ts: 0x157CC0086FFE1E64 dt: 0x000000000EF84046 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28102 ptr: 15123 data: 0x874C0086 module:  7 information type:  4 information field: 0x000C0086 ts: 0x157CC0086E000000 OLD: ts: 0x157CC0089E000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block: 28131 ptr: 10719 data: 0xC1D39F34 module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 40756  ts: 0x157CC0091D8F648E OLD: ts: 0x157CC0091D8F65BA dt: 0x000000000000012C tag: 3 type: 0
 ***            ADC data: block: 28132 ptr:  2867 data: 0xC1F28208 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33288  ts: 0x157CC0091AD74E6E OLD: ts: 0x157CC0091F88850E dt: 0x0000000004B136A0 tag: 3 type: 0
 ***            ADC data: block: 28135 ptr: 15555 data: 0xC1FF8155 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33109  ts: 0x157CC00933B91ACE OLD: ts: 0x157CC0093AE47BE6 dt: 0x00000000072B6118 tag: 2 type: 3
 ***            ADC data: block: 28151 ptr:  5247 data: 0xC1F28277 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33399  ts: 0x157CC009700041BE OLD: ts: 0x157CC00976FEEE5E dt: 0x0000000006FEACA0 tag: 3 type: 0
 ***            ADC data: block: 28152 ptr: 13315 data: 0xC1FF81BF module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33215  ts: 0x157CC00977C636CE OLD: ts: 0x157CC0097FF2F59E dt: 0x00000000082CBED0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28153 ptr:  6275 data: 0x874C0085 module:  7 information type:  4 information field: 0x000C0085 ts: 0x157CC0085A000000 OLD: ts: 0x157CC0097A000000 dt: 0x0000000120000000 tag: 2 type: 5
 ***            ADC data: block: 28153 ptr: 11291 data: 0xC1F98095 module:  7 fail: 0 range: 0 id:   505 channel: 57 adc: 32917  ts: 0x157CC00856FEEE5E OLD: ts: 0x157CC0085B71B29E dt: 0x000000000472C440 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28183 ptr: 11485 data: 0x874C009A module:  7 information type:  4 information field: 0x000C009A ts: 0x157CC009AE000000 OLD: ts: 0x157CC00A0E000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block: 28186 ptr:  2731 data: 0xC1F7808C module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32908  ts: 0x157CC009BEC57BFE OLD: ts: 0x157CC009BF9953AE dt: 0x0000000000D3D7B0 tag: 3 type: 0
 ***            ADC data: block: 28204 ptr:  2681 data: 0xC1F98240 module:  7 fail: 0 range: 0 id:   505 channel: 57 adc: 33344  ts: 0x157CC00A71737A4E OLD: ts: 0x157CC00A729B3114 dt: 0x000000000127B6C6 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 28205 ptr: 14397 data: 0x874C00A4 module:  7 information type:  4 information field: 0x000C00A4 ts: 0x157CC00A4E000000 OLD: ts: 0x157CC00A7E000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28210 ptr: 12689 data: 0x874C00A7 module:  7 information type:  4 information field: 0x000C00A7 ts: 0x157CC00A76000000 OLD: ts: 0x157CC00AA6000000 dt: 0x0000000030000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 28211 ptr:  5145 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157CC00A94BAD906 OLD: ts: 0x157CC00A9FBD4CFE dt: 0x000000000B0273F8 tag: 3 type: 0
 ***            ADC data: block: 28237 ptr: 11165 data: 0xC1F2811C module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33052  ts: 0x157CC00B04A9352E OLD: ts: 0x157CC00B054C45C0 dt: 0x0000000000A31092 tag: 2 type: 6
 ***            ADC data: block: 28242 ptr:  1593 data: 0xC1CD8154 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33108  ts: 0x157CC00B23E2947E OLD: ts: 0x157CC00B2A9B7FF8 dt: 0x0000000006B8EB7A tag: 2 type: 3
 ***   SYNC100 timestamp: block: 28242 ptr: 15553 data: 0x874C00AA module:  7 information type:  4 information field: 0x000C00AA ts: 0x157CC00AA8000000 OLD: ts: 0x157CC00B28000000 dt: 0x0000000080000000 tag: 2 type: 5
 ***            ADC data: block: 28242 ptr: 15941 data: 0xC1CD8174 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33140  ts: 0x157CC00AA5522ACE OLD: ts: 0x157CC00AA80FC36E dt: 0x0000000002BD98A0 tag: 3 type: 0
 ***            ADC data: block: 28243 ptr: 13919 data: 0xC1CA8056 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32854  ts: 0x157CC00AA980EFFE OLD: ts: 0x157CC00AAF74974E dt: 0x0000000005F3A750 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28243 ptr: 15257 data: 0x874C00A3 module:  7 information type:  4 information field: 0x000C00A3 ts: 0x157CC00A3A000000 OLD: ts: 0x157CC00AAA000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28245 ptr: 11063 data: 0x874C0086 module:  7 information type:  4 information field: 0x000C0086 ts: 0x157CC00860004B0E OLD: ts: 0x157CC00A40004B0E dt: 0x00000001E0000000 tag: 2 type: 5
 ***            ADC data: block: 28246 ptr:  7857 data: 0xC1F780B3 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32947  ts: 0x157CC00863D8EB6E OLD: ts: 0x157CC008641A9DDE dt: 0x000000000041B270 tag: 3 type: 0
 ***            ADC data: block: 28261 ptr:  2289 data: 0xC1DC80D4 module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32980  ts: 0x157CC00B748B05DE OLD: ts: 0x157CC00B748B131C dt: 0x0000000000000D3E tag: 3 type: 0
 ***            ADC data: block: 28282 ptr:  1209 data: 0xC1DA9E22 module:  7 fail: 0 range: 0 id:   474 channel: 26 adc: 40482  ts: 0x157CC00BCD6AF0AE OLD: ts: 0x157CC00BCD7D39BE dt: 0x0000000000124910 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28282 ptr:  9677 data: 0x874C00B5 module:  7 information type:  4 information field: 0x000C00B5 ts: 0x157CC00B5000030E OLD: ts: 0x157CC00BC000030E dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28285 ptr:  1499 data: 0x874C00B3 module:  7 information type:  4 information field: 0x000C00B3 ts: 0x157CC00B3E000000 OLD: ts: 0x157CC00B5E000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block: 28308 ptr: 15083 data: 0xC1E18054 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32852  ts: 0x157CC00C5B7BA8AE OLD: ts: 0x157CC00C5B8820B6 dt: 0x00000000000C7808 tag: 2 type: 6
 ***            ADC data: block: 28309 ptr: 15927 data: 0xC1CD81E6 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33254  ts: 0x157CC00C57670C3E OLD: ts: 0x157CC00C5F21AE0E dt: 0x0000000007BAA1D0 tag: 3 type: 0
 ***            ADC data: block: 28310 ptr: 13903 data: 0xC1F48113 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33043  ts: 0x157CC00C5159F90E OLD: ts: 0x157CC00C5BCECB5E dt: 0x000000000A74D250 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28312 ptr:  4221 data: 0x874C00BE module:  7 information type:  4 information field: 0x000C00BE ts: 0x157CC00BE8000000 OLD: ts: 0x157CC00C58000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28339 ptr: 10703 data: 0x874C00C9 module:  7 information type:  4 information field: 0x000C00C9 ts: 0x157CC00C9000067E OLD: ts: 0x157CC00CF000067E dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block: 28341 ptr:  2459 data: 0xC1F18162 module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33122  ts: 0x157CC00C99CF512E OLD: ts: 0x157CC00C9EF4798E dt: 0x0000000005252860 tag: 3 type: 0
 ***            ADC data: block: 28342 ptr:   435 data: 0xC1EF80F8 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33016  ts: 0x157CC00C906F7BFE OLD: ts: 0x157CC00C9E1FC75E dt: 0x000000000DB04B60 tag: 3 type: 0
 ***            ADC data: block: 28353 ptr: 12813 data: 0xC1EC817D module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33149  ts: 0x157CC00D330C1B6E OLD: ts: 0x157CC00D33E5012C dt: 0x0000000000D8E5BE tag: 2 type: 6
 ***            ADC data: block: 28354 ptr: 14869 data: 0xC1EF8127 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33063  ts: 0x157CC00D333CC9CE OLD: ts: 0x157CC00D374F52FE dt: 0x0000000004128930 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28356 ptr:  4059 data: 0x874C00D0 module:  7 information type:  4 information field: 0x000C00D0 ts: 0x157CC00D0A000000 OLD: ts: 0x157CC00D3A000000 dt: 0x0000000030000000 tag: 2 type: 5
 ***            ADC data: block: 28386 ptr: 15625 data: 0xC1E59FE9 module:  7 fail: 0 range: 0 id:   485 channel: 37 adc: 40937  ts: 0x157CC00DD676CB1E OLD: ts: 0x157CC00DD676CC4A dt: 0x000000000000012C tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28389 ptr:  5513 data: 0x874C00D8 module:  7 information type:  4 information field: 0x000C00D8 ts: 0x157CC00D8C000000 OLD: ts: 0x157CC00DEC000000 dt: 0x0000000060000000 tag: 2 type: 5
 ***            ADC data: block: 28390 ptr:  1485 data: 0xC1DE80A6 module:  7 fail: 0 range: 0 id:   478 channel: 30 adc: 32934  ts: 0x157CC00D89F06C2E OLD: ts: 0x157CC00D8FB21FEE dt: 0x0000000005C1B3C0 tag: 3 type: 0
 *** FEE64 discriminator: block: 28390 ptr: 15833 data: 0x84640010 module:  4 information type:  6 information field: 0x00040010 ts: 0x157CC00D83E8D15C OLD: ts: 0x157CC00D8E3691BE dt: 0x000000000A4DC062 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28400 ptr:  8673 data: 0x874C00DF module:  7 information type:  4 information field: 0x000C00DF ts: 0x157CC00DF0000000 OLD: ts: 0x157CC00E00000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28401 ptr: 11305 data: 0x874C00D8 module:  7 information type:  4 information field: 0x000C00D8 ts: 0x157CC00D80007C4E OLD: ts: 0x157CC00DF0007C4E dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 28424 ptr: 12857 data: 0xC1FF81C0 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33216  ts: 0x157CC00E8DF2A7FE OLD: ts: 0x157CC00E8F09959E dt: 0x000000000116EDA0 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 28426 ptr:  9793 data: 0x874C00E5 module:  7 information type:  4 information field: 0x000C00E5 ts: 0x157CC00E500041FE OLD: ts: 0x157CC00E900041FE dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block: 28432 ptr: 10443 data: 0xC1C58110 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33040  ts: 0x157CC00EB002D23E OLD: ts: 0x157CC00EB9A721FE dt: 0x0000000009A44FC0 tag: 3 type: 0
 ***            ADC data: block: 28440 ptr: 14043 data: 0xC1F78065 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32869  ts: 0x157CC00EF08D2DEE OLD: ts: 0x157CC00EF0D4140C dt: 0x000000000046E61E tag: 2 type: 4
 ***   SYNC100 timestamp: block: 28443 ptr:  1179 data: 0x874C00EE module:  7 information type:  4 information field: 0x000C00EE ts: 0x157CC00EE0000000 OLD: ts: 0x157CC00EF0000000 dt: 0x0000000010000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28462 ptr: 15349 data: 0x874C00F5 module:  7 information type:  4 information field: 0x000C00F5 ts: 0x157CC00F500741CE OLD: ts: 0x157CC00F700741CE dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block: 28510 ptr:  3343 data: 0xC1C5812B module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33067  ts: 0x157CC0102057428E OLD: ts: 0x157CC01021333EC4 dt: 0x0000000000DBFC36 tag: 2 type: 6
 ***            ADC data: block: 28514 ptr: 12639 data: 0xC1FD8113 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33043  ts: 0x157CC010414E505E OLD: ts: 0x157CC010456D3D6E dt: 0x00000000041EED10 tag: 3 type: 0
 ***            ADC data: block: 28515 ptr: 10615 data: 0xC1FF81B0 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33200  ts: 0x157CC01042CF7AAE OLD: ts: 0x157CC010457C69AE dt: 0x0000000002ACEF00 tag: 3 type: 0
 ***            ADC data: block: 28527 ptr: 12181 data: 0xC1C581CC module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33228  ts: 0x157CC01077052BBE OLD: ts: 0x157CC01077FBC064 dt: 0x0000000000F694A6 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 28528 ptr: 14647 data: 0x874C0105 module:  7 information type:  4 information field: 0x000C0105 ts: 0x157CC010500102FE OLD: ts: 0x157CC010700102FE dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block: 28531 ptr:  1637 data: 0xC1CD813A module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33082  ts: 0x157CC01059B501AE OLD: ts: 0x157CC0105AF5A35E dt: 0x000000000140A1B0 tag: 3 type: 0
 ***            ADC data: block: 28560 ptr:  1409 data: 0xC1F3800C module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32780  ts: 0x157CC0110D37A40E OLD: ts: 0x157CC0110E555818 dt: 0x00000000011DB40A tag: 2 type: 6
 ***            ADC data: block: 28560 ptr: 15051 data: 0xC1C58014 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32788  ts: 0x157CC01110009A8E OLD: ts: 0x157CC0111011685E dt: 0x000000000010CDD0 tag: 3 type: 0
 ***            ADC data: block: 28561 ptr: 11959 data: 0xC1E18094 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32916  ts: 0x157CC01110119F0E OLD: ts: 0x157CC01113DAA06E dt: 0x0000000003C90160 tag: 3 type: 0
 ***            ADC data: block: 28570 ptr: 15913 data: 0xC1F881E3 module:  7 fail: 0 range: 0 id:   504 channel: 56 adc: 33251  ts: 0x157CC011489F07AE OLD: ts: 0x157CC0114E210ABE dt: 0x0000000005820310 tag: 3 type: 0
 ***            ADC data: block: 28571 ptr: 13889 data: 0xC1F28142 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33090  ts: 0x157CC01143DB34DE OLD: ts: 0x157CC0114CE9C8EE dt: 0x00000000090E9410 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28572 ptr:  4773 data: 0x874C010B module:  7 information type:  4 information field: 0x000C010B ts: 0x157CC010B6000000 OLD: ts: 0x157CC01146000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***            ADC data: block: 28626 ptr: 15761 data: 0xC1F481C3 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33219  ts: 0x157CC0125230E0CE OLD: ts: 0x157CC0125241D0A0 dt: 0x000000000010EFD2 tag: 2 type: 6
 ***            ADC data: block: 28631 ptr:  4487 data: 0xC1C08108 module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33032  ts: 0x157CC01278ED8C7E OLD: ts: 0x157CC0127A68DD1C dt: 0x00000000017B509E tag: 2 type: 3
 ***   SYNC100 timestamp: block: 28632 ptr: 12555 data: 0x874C011B module:  7 information type:  4 information field: 0x000C011B ts: 0x157CC011B0000000 OLD: ts: 0x157CC01270000000 dt: 0x00000000C0000000 tag: 2 type: 5
 ***            ADC data: block: 28650 ptr:  9767 data: 0xC1DB817A module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33146  ts: 0x157CC012C2A70A0E OLD: ts: 0x157CC012C2B5DAE8 dt: 0x00000000000ED0DA tag: 2 type: 6
 ***   SYNC100 timestamp: block: 28654 ptr:  3785 data: 0x874C0128 module:  7 information type:  4 information field: 0x000C0128 ts: 0x157CC0128C000000 OLD: ts: 0x157CC012DC000000 dt: 0x0000000050000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 28654 ptr:  4955 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157CC012877D0A6E OLD: ts: 0x157CC0128C614D2E dt: 0x0000000004E442C0 tag: 3 type: 0
 ***            ADC data: block: 28667 ptr: 14595 data: 0xC1E1808F module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32911  ts: 0x157CC01323C4CABE OLD: ts: 0x157CC0132BF69A7E dt: 0x000000000831CFC0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28668 ptr:  6095 data: 0x874C012E module:  7 information type:  4 information field: 0x000C012E ts: 0x157CC012E6000000 OLD: ts: 0x157CC01326000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block: 28685 ptr: 15739 data: 0xC1FF814D module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33101  ts: 0x157CC013A0EDD51E OLD: ts: 0x157CC013AAAE06AE dt: 0x0000000009C03190 tag: 3 type: 0
 ***            ADC data: block: 28736 ptr:  4071 data: 0xC1EC8180 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33152  ts: 0x157CC0145B3FDE8E OLD: ts: 0x157CC0145C2CFB60 dt: 0x0000000000ED1CD2 tag: 2 type: 6
 ***            ADC data: block: 28737 ptr:   207 data: 0xC1F28199 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33177  ts: 0x157CC01452A973DE OLD: ts: 0x157CC0145E11ABCE dt: 0x000000000B6837F0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28738 ptr: 10115 data: 0x874C0129 module:  7 information type:  4 information field: 0x000C0129 ts: 0x157CC012900016FE OLD: ts: 0x157CC014500016FE dt: 0x00000001C0000000 tag: 2 type: 5
 ***            ADC data: block: 28741 ptr: 13591 data: 0xC1F37F9F module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32671  ts: 0x157CC0147088CF7E OLD: ts: 0x157CC0147BA85E4E dt: 0x000000000B1F8ED0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28742 ptr:  8655 data: 0x874C00D1 module:  7 information type:  4 information field: 0x000C00D1 ts: 0x157CC00D14000000 OLD: ts: 0x157CC01474000000 dt: 0x0000000760000000 tag: 2 type: 5
 ***            ADC data: block: 28742 ptr: 11569 data: 0xC1F280DE module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32990  ts: 0x157CC00D14D3D70E OLD: ts: 0x157CC00D1E9F982E dt: 0x0000000009CBC120 tag: 3 type: 0
 *** FEE64 discriminator: block: 28743 ptr:  9545 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157CC00D184967CA OLD: ts: 0x157CC00D19010DCE dt: 0x0000000000B7A604 tag: 3 type: 0
 ***            ADC data: block: 28759 ptr: 13441 data: 0xC1C38150 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33104  ts: 0x157CC014C3FA243E OLD: ts: 0x157CC014C88C6F6E dt: 0x0000000004924B30 tag: 3 type: 0
 ***            ADC data: block: 28772 ptr: 12269 data: 0xC1EF7F80 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32640  ts: 0x157CC01508197C8E OLD: ts: 0x157CC015098CB80E dt: 0x0000000001733B80 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28773 ptr: 13989 data: 0x874C0149 module:  7 information type:  4 information field: 0x000C0149 ts: 0x157CC0149E000000 OLD: ts: 0x157CC0150E000000 dt: 0x0000000070000000 tag: 2 type: 5
 ***            ADC data: block: 28782 ptr:  7715 data: 0xC1D07F61 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32609  ts: 0x157CC015430F9C7E OLD: ts: 0x157CC0154AA22DDE dt: 0x0000000007929160 tag: 3 type: 0
 ***            ADC data: block: 28783 ptr:  5691 data: 0xC1EF8200 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33280  ts: 0x157CC015433F3A0E OLD: ts: 0x157CC01547227D0E dt: 0x0000000003E34300 tag: 3 type: 0
 ***            ADC data: block: 28803 ptr: 11395 data: 0xC1C18123 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33059  ts: 0x157CC015BA0998BE OLD: ts: 0x157CC015BAF891A8 dt: 0x0000000000EEF8EA tag: 2 type: 6
 ***            ADC data: block: 28825 ptr:  8061 data: 0xC1D78044 module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32836  ts: 0x157CC0161514060E OLD: ts: 0x157CC016156AA1A8 dt: 0x0000000000569B9A tag: 2 type: 6
 ***   SYNC100 timestamp: block: 28826 ptr:  2951 data: 0x874C0158 module:  7 information type:  4 information field: 0x000C0158 ts: 0x157CC0158E000000 OLD: ts: 0x157CC0161E000000 dt: 0x0000000090000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28827 ptr:  8643 data: 0x874C0158 module:  7 information type:  4 information field: 0x000C0158 ts: 0x157CC0158005F14E OLD: ts: 0x157CC0159005F14E dt: 0x0000000010000000 tag: 2 type: 5
 ***            ADC data: block: 28854 ptr:  7201 data: 0xC1D580B9 module:  7 fail: 0 range: 0 id:   469 channel: 21 adc: 32953  ts: 0x157CC016AB7C99EE OLD: ts: 0x157CC016ABC06DEA dt: 0x000000000043D3FC tag: 2 type: 6
 ***            ADC data: block: 28855 ptr:  5583 data: 0xC1FF823A module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33338  ts: 0x157CC016A000042E OLD: ts: 0x157CC016AE8BDD8E dt: 0x000000000E8BD960 tag: 3 type: 0
 ***            ADC data: block: 28874 ptr:  2089 data: 0xC1DC80B9 module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32953  ts: 0x157CC0170924669E OLD: ts: 0x157CC01709DF0EEA dt: 0x0000000000BAA84C tag: 2 type: 6
 ***            ADC data: block: 28874 ptr: 14643 data: 0xC1C580AB module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32939  ts: 0x157CC017000C098E OLD: ts: 0x157CC0170C159AEE dt: 0x000000000C099160 tag: 3 type: 0
 ***            ADC data: block: 28908 ptr: 11901 data: 0xC1E18145 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33093  ts: 0x157CC017B355427E OLD: ts: 0x157CC017B9EFE4EE dt: 0x00000000069AA270 tag: 2 type: 3
 ***            ADC data: block: 28909 ptr:  9877 data: 0xC1E180A8 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32936  ts: 0x157CC017B3FD7F6E OLD: ts: 0x157CC017B7D7ADBE dt: 0x0000000003DA2E50 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28910 ptr:  7667 data: 0x874C016D module:  7 information type:  4 information field: 0x000C016D ts: 0x157CC016D8000000 OLD: ts: 0x157CC017B8000000 dt: 0x00000000E0000000 tag: 2 type: 5
 ***            ADC data: block: 28926 ptr: 13073 data: 0xC1F38023 module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32803  ts: 0x157CC017FC4611CE OLD: ts: 0x157CC017FD1D3550 dt: 0x0000000000D72382 tag: 2 type: 6
 ***            ADC data: block: 28927 ptr:  6799 data: 0xC1FD81C0 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33216  ts: 0x157CC017F261640E OLD: ts: 0x157CC017FE8AC9DE dt: 0x000000000C2965D0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28932 ptr:   995 data: 0x874C017C module:  7 information type:  4 information field: 0x000C017C ts: 0x157CC017C8000000 OLD: ts: 0x157CC01828000000 dt: 0x0000000060000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 28932 ptr: 10857 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157CC017C3022C3A OLD: ts: 0x157CC017CA775E1E dt: 0x00000000077531E4 tag: 3 type: 0
 ***            ADC data: block: 28949 ptr: 14139 data: 0xC1DB81A3 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33187  ts: 0x157CC0186947F40E OLD: ts: 0x157CC01869E5BE6E dt: 0x00000000009DCA60 tag: 2 type: 6
 ***            ADC data: block: 28957 ptr:  8189 data: 0xC1F18058 module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 32856  ts: 0x157CC0188678D2BE OLD: ts: 0x157CC018873AA1DC dt: 0x0000000000C1CF1E tag: 2 type: 6
 ***            ADC data: block: 28958 ptr:  6197 data: 0xC1C58217 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33303  ts: 0x157CC018849A829E OLD: ts: 0x157CC0188C4C51EE dt: 0x0000000007B1CF50 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 28958 ptr: 10949 data: 0x874C0183 module:  7 information type:  4 information field: 0x000C0183 ts: 0x157CC01836000000 OLD: ts: 0x157CC01886000000 dt: 0x0000000050000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block: 28982 ptr: 15957 data: 0x874C018C module:  7 information type:  4 information field: 0x000C018C ts: 0x157CC018C00006CE OLD: ts: 0x157CC019000006CE dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block: 28984 ptr:  6215 data: 0xC1D0804F module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32847  ts: 0x157CC018C2DD2EFE OLD: ts: 0x157CC018C71E141E dt: 0x000000000440E520 tag: 3 type: 0
 ***            ADC data: block: 29032 ptr: 16363 data: 0xC1DB81CD module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33229  ts: 0x157CC019E380437E OLD: ts: 0x157CC019E3935FFE dt: 0x0000000000131C80 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 29038 ptr:  3901 data: 0x874C0192 module:  7 information type:  4 information field: 0x000C0192 ts: 0x157CC01926000000 OLD: ts: 0x157CC01A16000000 dt: 0x00000000F0000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 29038 ptr: 13831 data: 0x80641010 module:  0 information type:  6 information field: 0x00041010 ts: 0x157CC01922069626 OLD: ts: 0x157CC0192897D82E dt: 0x0000000006914208 tag: 3 type: 0
 ***            ADC data: block: 29061 ptr: 14855 data: 0xC1C581D3 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33235  ts: 0x157CC01A68BD843E OLD: ts: 0x157CC01A68CEF2B4 dt: 0x0000000000116E76 tag: 2 type: 6
 ***   SYNC100 timestamp: block: 29062 ptr: 14391 data: 0x874C01A2 module:  7 information type:  4 information field: 0x000C01A2 ts: 0x157CC01A20000000 OLD: ts: 0x157CC01A60000000 dt: 0x0000000040000000 tag: 2 type: 5
 ***            ADC data: block: 29066 ptr:  6653 data: 0xC1F78076 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32886  ts: 0x157CC01A807BA65E OLD: ts: 0x157CC01A8FEF5E60 dt: 0x000000000F73B802 tag: 3 type: 0
 ***            ADC data: block: 29086 ptr:   167 data: 0xC1F98191 module:  7 fail: 0 range: 0 id:   505 channel: 57 adc: 33169  ts: 0x157CC01AD613B09E OLD: ts: 0x157CC01AD6CF3F44 dt: 0x0000000000BB8EA6 tag: 2 type: 6
 ***            ADC data: block: 29089 ptr: 14537 data: 0xC1EF81F6 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33270  ts: 0x157CC01AF47960DE OLD: ts: 0x157CC01AF6260FF8 dt: 0x0000000001ACAF1A tag: 2 type: 3
 ***   SYNC100 timestamp: block: 29090 ptr:  9643 data: 0x874C01A9 module:  7 information type:  4 information field: 0x000C01A9 ts: 0x157CC01A98000000 OLD: ts: 0x157CC01AF8000000 dt: 0x0000000060000000 tag: 2 type: 5
 *** FEE64 discriminator: block: 29090 ptr: 12513 data: 0x80640010 module:  0 information type:  6 information field: 0x00040010 ts: 0x157CC01A96962B6C OLD: ts: 0x157CC01A98ED7C0E dt: 0x00000000025750A2 tag: 3 type: 0
 ***            ADC data: block: 29117 ptr:  3063 data: 0xC1C5806B module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32875  ts: 0x157CC01B5989ECFE OLD: ts: 0x157CC01B599603E0 dt: 0x00000000000C16E2 tag: 2 type: 6
 ***            ADC data: block: 29118 ptr:   215 data: 0xC1C181BD module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33213  ts: 0x157CC01B5375D2CE OLD: ts: 0x157CC01B5C0B86FE dt: 0x000000000895B430 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 29118 ptr:  8891 data: 0x874C01B0 module:  7 information type:  4 information field: 0x000C01B0 ts: 0x157CC01B06000000 OLD: ts: 0x157CC01B56000000 dt: 0x0000000050000000 tag: 2 type: 5
 ***            ADC data: block: 29119 ptr: 12539 data: 0xC1C58050 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32848  ts: 0x157CC01B0C0BBDAE OLD: ts: 0x157CC01B0CFE475E dt: 0x0000000000F289B0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 29134 ptr: 11471 data: 0x874C01B9 module:  7 information type:  4 information field: 0x000C01B9 ts: 0x157CC01B9C000000 OLD: ts: 0x157CC01BBC000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***            ADC data: block: 29134 ptr: 15777 data: 0xC1D28058 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32856  ts: 0x157CC01B9C65BB4E OLD: ts: 0x157CC01B9D409CAE dt: 0x0000000000DAE160 tag: 3 type: 0
 ***            ADC data: block: 29167 ptr: 13003 data: 0xC1D4813F module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33087  ts: 0x157CC01C525F7E8E OLD: ts: 0x157CC01C52E9659A dt: 0x000000000089E70C tag: 2 type: 6
 ***            ADC data: block: 29189 ptr:  5177 data: 0xC1F280D2 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32978  ts: 0x157CC01CA463EC7E OLD: ts: 0x157CC01CA4CC5A34 dt: 0x0000000000686DB6 tag: 2 type: 6
 ***            ADC data: block: 29190 ptr:   969 data: 0xC1D07F92 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32658  ts: 0x157CC01CA02EF56E OLD: ts: 0x157CC01CA7118FEE dt: 0x0000000006E29A80 tag: 3 type: 0
 ***            ADC data: block: 29191 ptr: 13295 data: 0xC1FD8229 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33321  ts: 0x157CC01CA1BF611E OLD: ts: 0x157CC01CAC8763BE dt: 0x000000000AC802A0 tag: 3 type: 0
 ***   SYNC100 timestamp: block: 29192 ptr:  4599 data: 0x874C0158 module:  7 information type:  4 information field: 0x000C0158 ts: 0x157CC01584000000 OLD: ts: 0x157CC01CA4000000 dt: 0x0000000720000000 tag: 2 type: 5
 ***            ADC data: block: 29218 ptr: 14805 data: 0xC1EC8154 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33108  ts: 0x157CC01D327C443E OLD: ts: 0x157CC01D3C79299E dt: 0x0000000009FCE560 tag: 3 type: 0
 ***            ADC data: block: 29231 ptr:  7513 data: 0xC1F480D1 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32977  ts: 0x157CC01D8215095E OLD: ts: 0x157CC01D8CE78EEE dt: 0x000000000AD28590 tag: 3 type: 0
 ***            ADC data: block: 29248 ptr:  9151 data: 0xC1E97FD7 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32727  ts: 0x157CC01DF21BFDDE OLD: ts: 0x157CC01DFBDAC97E dt: 0x0000000009BECBA0 tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:      29273
          ADC data format:  223421345 (    3339.1 Hz)
        Other data format:   16178188 (     241.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:        870 (       0.0 Hz)
                              SYNC100:     320561 (       4.8 Hz)
                              WR48-63:     320637 (       4.8 Hz)
                           FEE64 disc:   15536120 (     232.2 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:    1717905 (      25.7 Hz)

                Timewarps:        ADC:       1525 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:         44 (       0.0 Hz)
                              SYNC100:        764 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:        193 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:    66910.602 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000           669105.573
  2                0.000           669105.573
  3                0.000           669105.573
  4                0.000           669127.383
  5                0.000           669105.573
  6                0.000           669105.573
  7                0.000           716741.763
  8                0.000           669105.573
  9                0.000           669105.573
 10                0.000           669105.573
 11                0.000           669105.573
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6110127    1513310          0          0          0          5      27877      27884    1457544          0          0     390222
  1    5070663     987935          0          0          0          4      26592      26596     934743          0          0          0
  2    3587479     799419          0          0          0          4      26590      26597     746228          0          0          0
  3    6781811    1287563          0          0          0          4      28055      28064    1231440          0          0     124190
  4    7066991    1323016          0          0          0          6      27815      27820    1267375          0          0     588307
  5    6847028     964420          0          0          0          4      26555      26560     911301          0          0          0
  6    3189015     743727          0          0          0          4      26559      26570     690594          0          0          0
  7  161819968    4257643          0          0          0        822      21607      21610    4213604          0          0       5502
  8    7109956    1225249          0          0          0          5      27840      27844    1169560          0          0     559025
  9    5135479    1016932          0          0          0          4      26560      26565     963803          0          0          1
 10    3666556     759183          0          0          0          4      26561      26569     706049          0          0          0
 11    7036272    1299791          0          0          0          4      27950      27958    1243879          0          0      50658
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          1          0          0          2          0        119          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          4          0          0          0
  3          1          0          0          0          0          0          0          0          0
  4          2          0          0          1          0         26          0          0          0
  5          2          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7       1519          0         44        761          0         29          0          0          0
  8          0          0          0          0          0         14          0          0          0
  9          0          0          0          0          0          1          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
... 25 more lines ...
Attachment 2: WR_sample_R1_0_24jan19.log
[td@localhost GSIJan19]$ ~/analyser/analyser v R1_0 | more
 *** TDR format 3.3.0 analyser - TD - May 2018
  verbose
 ***            ADC data: block:     1 ptr:     7 data: 0xC1D07FBF module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32703  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:     9 data: 0xC1E0816B module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33131  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:    11 data: 0xC1F080C0 module:  7 fail: 0 range: 0 id:   496 channel: 48 adc: 32960  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:    13 data: 0xC1C580F6 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33014  ts: 0x00000000060A86BE
 ***            ADC data: block:     1 ptr:    15 data: 0xC1F48118 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33048  ts: 0x00000000060A86BE
 ***            ADC data: block:     1 ptr:    17 data: 0xC1C68136 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33078  ts: 0x00000000060A8E8E
 ***            ADC data: block:     1 ptr:    19 data: 0xC1FE8093 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 32915  ts: 0x00000000060A8E8E
 ***            ADC data: block:     1 ptr:    21 data: 0xC1CA8070 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32880  ts: 0x00000000060AA5FE
 ***            ADC data: block:     1 ptr:    23 data: 0xC1E880F8 module:  7 fail: 0 range: 0 id:   488 channel: 40 adc: 33016  ts: 0x00000000060AA5FE
 ***            ADC data: block:     1 ptr:    25 data: 0xC1CB8102 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33026  ts: 0x00000000060AADCE
 ***            ADC data: block:     1 ptr:    27 data: 0xC1EB815B module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33115  ts: 0x00000000060AADCE
 ***            ADC data: block:     1 ptr:    29 data: 0xC1D4817E module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33150  ts: 0x00000000060AB59E
 ***            ADC data: block:     1 ptr:    31 data: 0xC1FA815A module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 33114  ts: 0x00000000060AB59E
 ***            ADC data: block:     1 ptr:    33 data: 0xC1D38076 module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 32886  ts: 0x00000000060ACD0E
 ***            ADC data: block:     1 ptr:    35 data: 0xC1DD7F61 module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32609  ts: 0x00000000060AD4DE
 ***            ADC data: block:     1 ptr:    37 data: 0xC1FF8359 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33625  ts: 0x00000000060ADCAE
 ***            ADC data: block:     1 ptr:    39 data: 0xC1C480A1 module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32929  ts: 0x00000000060AF41E
 ***            ADC data: block:     1 ptr:    41 data: 0xC1F7806A module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32874  ts: 0x00000000060AF41E
 ***            ADC data: block:     1 ptr:    43 data: 0xC1C681EC module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33260  ts: 0x00000000060B1B2E
 ***            ADC data: block:     1 ptr:    45 data: 0xC1FE8162 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33122  ts: 0x00000000060B1B2E
 ***            ADC data: block:     1 ptr:    47 data: 0xC1EB80B2 module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 32946  ts: 0x00000000060B617E
 ***            ADC data: block:     1 ptr:    49 data: 0xC1DD7F96 module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32662  ts: 0x00000000060B694E
 ***            ADC data: block:     1 ptr:    51 data: 0xC1CB809A module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 32922  ts: 0x00000000060B80BE
 ***            ADC data: block:     1 ptr:    53 data: 0xC1F7800D module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32781  ts: 0x00000000060B80BE
 ***            ADC data: block:     1 ptr:    55 data: 0xC1C58139 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33081  ts: 0x00000000060BA7CE
 ***            ADC data: block:     1 ptr:    57 data: 0xC1F48145 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33093  ts: 0x00000000060BA7CE
 ***            ADC data: block:     1 ptr:    59 data: 0xC1C68108 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33032  ts: 0x00000000060BAF9E
 ***            ADC data: block:     1 ptr:    61 data: 0xC1FE80E8 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33000  ts: 0x00000000060BAF9E
 ***            ADC data: block:     1 ptr:    63 data: 0xC1CA8037 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32823  ts: 0x00000000060BC70E
 ***            ADC data: block:     1 ptr:    65 data: 0xC1EB81FE module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33278  ts: 0x00000000060BF5EE
 ***            ADC data: block:     1 ptr:    67 data: 0xC1DD7F7C module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32636  ts: 0x00000000060BFDBE
 ***            ADC data: block:     1 ptr:    69 data: 0xC1CB808D module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 32909  ts: 0x00000000060C0D5E
 ***            ADC data: block:     1 ptr:    71 data: 0xC1E68050 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32848  ts: 0x00000000060C0D5E
 ***            ADC data: block:     1 ptr:    73 data: 0xC1E97F94 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32660  ts: 0x00000000060C152E
 ***            ADC data: block:     1 ptr:    75 data: 0xC1F680EC module:  7 fail: 0 range: 0 id:   502 channel: 54 adc: 33004  ts: 0x00000000060C152E
 ***            ADC data: block:     1 ptr:    77 data: 0xC1F78092 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32914  ts: 0x00000000060C1CFE
 ***            ADC data: block:     1 ptr:    79 data: 0xC1C581C2 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33218  ts: 0x00000000060C346E
 ***            ADC data: block:     1 ptr:    81 data: 0xC1F481C0 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33216  ts: 0x00000000060C346E
 ***            ADC data: block:     1 ptr:    83 data: 0xC1C6812E module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33070  ts: 0x00000000060C3C3E
 ***            ADC data: block:     1 ptr:    85 data: 0xC1FE80F5 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33013  ts: 0x00000000060C3C3E
 ***            ADC data: block:     1 ptr:    87 data: 0xC1C48085 module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32901  ts: 0x00000000060C634E
 ***            ADC data: block:     1 ptr:    89 data: 0xC1F28177 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33143  ts: 0x00000000060C828E
 ***            ADC data: block:     1 ptr:    91 data: 0xC1D7801A module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32794  ts: 0x00000000060CA1CE
 ***            ADC data: block:     1 ptr:    93 data: 0xC1C780CB module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32971  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    95 data: 0xC1E880FF module:  7 fail: 0 range: 0 id:   488 channel: 40 adc: 33023  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    97 data: 0xC1FB80FA module:  7 fail: 0 range: 0 id:   507 channel: 59 adc: 33018  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    99 data: 0xC1E97FE9 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32745  ts: 0x00000000060CB16E
 ***            ADC data: block:     1 ptr:   101 data: 0xC1EF8131 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33073  ts: 0x00000000060CC8DE
 ***            ADC data: block:     1 ptr:   103 data: 0xC1C57FFA module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32762  ts: 0x00000000060CD0AE
 ***            ADC data: block:     1 ptr:   105 data: 0xC1F48060 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32864  ts: 0x00000000060CD0AE
 ***            ADC data: block:     1 ptr:   107 data: 0xC1C6815D module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33117  ts: 0x00000000060CD87E

[td@localhost GSIJan19]$ 
[td@localhost GSIJan19]$ ~/analyser/analyser v R1_0 | more
 *** TDR format 3.3.0 analyser - TD - May 2018
  verbose
 ***            ADC data: block:     1 ptr:     7 data: 0xC1D07FBF module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32703  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:     9 data: 0xC1E0816B module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33131  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:    11 data: 0xC1F080C0 module:  7 fail: 0 range: 0 id:   496 channel: 48 adc: 32960  ts: 0x00000000060A6F4E
 ***            ADC data: block:     1 ptr:    13 data: 0xC1C580F6 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33014  ts: 0x00000000060A86BE
 ***            ADC data: block:     1 ptr:    15 data: 0xC1F48118 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33048  ts: 0x00000000060A86BE
 ***            ADC data: block:     1 ptr:    17 data: 0xC1C68136 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33078  ts: 0x00000000060A8E8E
 ***            ADC data: block:     1 ptr:    19 data: 0xC1FE8093 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 32915  ts: 0x00000000060A8E8E
 ***            ADC data: block:     1 ptr:    21 data: 0xC1CA8070 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32880  ts: 0x00000000060AA5FE
 ***            ADC data: block:     1 ptr:    23 data: 0xC1E880F8 module:  7 fail: 0 range: 0 id:   488 channel: 40 adc: 33016  ts: 0x00000000060AA5FE
 ***            ADC data: block:     1 ptr:    25 data: 0xC1CB8102 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33026  ts: 0x00000000060AADCE
 ***            ADC data: block:     1 ptr:    27 data: 0xC1EB815B module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33115  ts: 0x00000000060AADCE
 ***            ADC data: block:     1 ptr:    29 data: 0xC1D4817E module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33150  ts: 0x00000000060AB59E
 ***            ADC data: block:     1 ptr:    31 data: 0xC1FA815A module:  7 fail: 0 range: 0 id:   506 channel: 58 adc: 33114  ts: 0x00000000060AB59E
 ***            ADC data: block:     1 ptr:    33 data: 0xC1D38076 module:  7 fail: 0 range: 0 id:   467 channel: 19 adc: 32886  ts: 0x00000000060ACD0E
 ***            ADC data: block:     1 ptr:    35 data: 0xC1DD7F61 module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32609  ts: 0x00000000060AD4DE
 ***            ADC data: block:     1 ptr:    37 data: 0xC1FF8359 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33625  ts: 0x00000000060ADCAE
 ***            ADC data: block:     1 ptr:    39 data: 0xC1C480A1 module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32929  ts: 0x00000000060AF41E
 ***            ADC data: block:     1 ptr:    41 data: 0xC1F7806A module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32874  ts: 0x00000000060AF41E
 ***            ADC data: block:     1 ptr:    43 data: 0xC1C681EC module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33260  ts: 0x00000000060B1B2E
 ***            ADC data: block:     1 ptr:    45 data: 0xC1FE8162 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33122  ts: 0x00000000060B1B2E
 ***            ADC data: block:     1 ptr:    47 data: 0xC1EB80B2 module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 32946  ts: 0x00000000060B617E
 ***            ADC data: block:     1 ptr:    49 data: 0xC1DD7F96 module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32662  ts: 0x00000000060B694E
 ***            ADC data: block:     1 ptr:    51 data: 0xC1CB809A module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 32922  ts: 0x00000000060B80BE
 ***            ADC data: block:     1 ptr:    53 data: 0xC1F7800D module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32781  ts: 0x00000000060B80BE
 ***            ADC data: block:     1 ptr:    55 data: 0xC1C58139 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33081  ts: 0x00000000060BA7CE
 ***            ADC data: block:     1 ptr:    57 data: 0xC1F48145 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33093  ts: 0x00000000060BA7CE
 ***            ADC data: block:     1 ptr:    59 data: 0xC1C68108 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33032  ts: 0x00000000060BAF9E
 ***            ADC data: block:     1 ptr:    61 data: 0xC1FE80E8 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33000  ts: 0x00000000060BAF9E
 ***            ADC data: block:     1 ptr:    63 data: 0xC1CA8037 module:  7 fail: 0 range: 0 id:   458 channel: 10 adc: 32823  ts: 0x00000000060BC70E
 ***            ADC data: block:     1 ptr:    65 data: 0xC1EB81FE module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33278  ts: 0x00000000060BF5EE
 ***            ADC data: block:     1 ptr:    67 data: 0xC1DD7F7C module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32636  ts: 0x00000000060BFDBE
 ***            ADC data: block:     1 ptr:    69 data: 0xC1CB808D module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 32909  ts: 0x00000000060C0D5E
 ***            ADC data: block:     1 ptr:    71 data: 0xC1E68050 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32848  ts: 0x00000000060C0D5E
 ***            ADC data: block:     1 ptr:    73 data: 0xC1E97F94 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32660  ts: 0x00000000060C152E
 ***            ADC data: block:     1 ptr:    75 data: 0xC1F680EC module:  7 fail: 0 range: 0 id:   502 channel: 54 adc: 33004  ts: 0x00000000060C152E
 ***            ADC data: block:     1 ptr:    77 data: 0xC1F78092 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32914  ts: 0x00000000060C1CFE
 ***            ADC data: block:     1 ptr:    79 data: 0xC1C581C2 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33218  ts: 0x00000000060C346E
 ***            ADC data: block:     1 ptr:    81 data: 0xC1F481C0 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33216  ts: 0x00000000060C346E
 ***            ADC data: block:     1 ptr:    83 data: 0xC1C6812E module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33070  ts: 0x00000000060C3C3E
 ***            ADC data: block:     1 ptr:    85 data: 0xC1FE80F5 module:  7 fail: 0 range: 0 id:   510 channel: 62 adc: 33013  ts: 0x00000000060C3C3E
 ***            ADC data: block:     1 ptr:    87 data: 0xC1C48085 module:  7 fail: 0 range: 0 id:   452 channel:  4 adc: 32901  ts: 0x00000000060C634E
 ***            ADC data: block:     1 ptr:    89 data: 0xC1F28177 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33143  ts: 0x00000000060C828E
 ***            ADC data: block:     1 ptr:    91 data: 0xC1D7801A module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32794  ts: 0x00000000060CA1CE
 ***            ADC data: block:     1 ptr:    93 data: 0xC1C780CB module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32971  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    95 data: 0xC1E880FF module:  7 fail: 0 range: 0 id:   488 channel: 40 adc: 33023  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    97 data: 0xC1FB80FA module:  7 fail: 0 range: 0 id:   507 channel: 59 adc: 33018  ts: 0x00000000060CA99E
 ***            ADC data: block:     1 ptr:    99 data: 0xC1E97FE9 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32745  ts: 0x00000000060CB16E
 ***            ADC data: block:     1 ptr:   101 data: 0xC1EF8131 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33073  ts: 0x00000000060CC8DE
 ***            ADC data: block:     1 ptr:   103 data: 0xC1C57FFA module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32762  ts: 0x00000000060CD0AE
 ***            ADC data: block:     1 ptr:   105 data: 0xC1F48060 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32864  ts: 0x00000000060CD0AE
 ***            ADC data: block:     1 ptr:   107 data: 0xC1C6815D module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33117  ts: 0x00000000060CD87E
/ 1894 ptr:
...skipping
 ***            ADC data: block:  1894 ptr:     7 data: 0xC1EC81F9 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33273  ts: 0x157C835E76CEFBBE
 ***            ADC data: block:  1894 ptr:     9 data: 0xC1F48108 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33032  ts: 0x157C835E76CEFBBE
 ***            ADC data: block:  1894 ptr:    11 data: 0xC1C580D6 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32982  ts: 0x157C835E76CF420E
 ***            ADC data: block:  1894 ptr:    13 data: 0xC1FF833D module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33597  ts: 0x157C835E76CF51AE
 ***            ADC data: block:  1894 ptr:    15 data: 0xC1F281C8 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33224  ts: 0x157C835E76CF78BE
 ***            ADC data: block:  1894 ptr:    17 data: 0xC1EC81D8 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33240  ts: 0x157C835E76CFAF6E
 ***            ADC data: block:  1894 ptr:    19 data: 0xC1F4810B module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33035  ts: 0x157C835E76CFAF6E
 ***            ADC data: block:  1894 ptr:    21 data: 0xC1C580C8 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32968  ts: 0x157C835E76CFD67E
 ***            ADC data: block:  1894 ptr:    23 data: 0xC1FF8309 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33545  ts: 0x157C835E76CFE61E
 ***            ADC data: block:  1894 ptr:    25 data: 0xC1C1819A module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33178  ts: 0x157C835E76D01CCE
 ***            ADC data: block:  1894 ptr:    27 data: 0xC1F48078 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32888  ts: 0x157C835E76D03C0E
 ***            ADC data: block:  1894 ptr:    29 data: 0xC1EC8160 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33120  ts: 0x157C835E76D043DE
 ***            ADC data: block:  1894 ptr:    31 data: 0xC1C58176 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33142  ts: 0x157C835E76D0825E
 ***            ADC data: block:  1894 ptr:    33 data: 0xC1FF82DE module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33502  ts: 0x157C835E76D091FE
 ***            ADC data: block:  1894 ptr:    35 data: 0xC1EC825D module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33373  ts: 0x157C835E76D0D84E
 ***            ADC data: block:  1894 ptr:    37 data: 0xC1F181BF module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33215  ts: 0x157C835E76D0F78E
 ***            ADC data: block:  1894 ptr:    39 data: 0xC1E97FE8 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32744  ts: 0x157C835E76D0FF5E
 ***            ADC data: block:  1894 ptr:    41 data: 0xC1F48125 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33061  ts: 0x157C835E76D0FF5E
 ***            ADC data: block:  1894 ptr:    43 data: 0xC1C18131 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33073  ts: 0x157C835E76D116CE
 ***            ADC data: block:  1894 ptr:    45 data: 0xC1E0820A module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33290  ts: 0x157C835E76D116CE
 ***            ADC data: block:  1894 ptr:    47 data: 0xC1FD81D4 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33236  ts: 0x157C835E76D116CE
 ***            ADC data: block:  1894 ptr:    49 data: 0xC1CB8127 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33063  ts: 0x157C835E76D11E9E
 ***            ADC data: block:  1894 ptr:    51 data: 0xC1D08050 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32848  ts: 0x157C835E76D11E9E
 ***            ADC data: block:  1894 ptr:    53 data: 0xC1E181AA module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33194  ts: 0x157C835E76D11E9E
 ***            ADC data: block:  1894 ptr:    55 data: 0xC1CD820A module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33290  ts: 0x157C835E76D1266E
 ***            ADC data: block:  1894 ptr:    57 data: 0xC1D681D3 module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 33235  ts: 0x157C835E76D1266E
 ***            ADC data: block:  1894 ptr:    59 data: 0xC1E38168 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33128  ts: 0x157C835E76D1266E
 ***            ADC data: block:  1894 ptr:    61 data: 0xC1D78044 module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32836  ts: 0x157C835E76D12E3E
 ***            ADC data: block:  1894 ptr:    63 data: 0xC1EA80AA module:  7 fail: 0 range: 0 id:   490 channel: 42 adc: 32938  ts: 0x157C835E76D12E3E
 ***            ADC data: block:  1894 ptr:    65 data: 0xC1DA8061 module:  7 fail: 0 range: 0 id:   474 channel: 26 adc: 32865  ts: 0x157C835E76D1360E
 ***            ADC data: block:  1894 ptr:    67 data: 0xC1EB813B module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33083  ts: 0x157C835E76D1360E
 ***            ADC data: block:  1894 ptr:    69 data: 0xC1C5819F module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33183  ts: 0x157C835E76D13DDE
 ***            ADC data: block:  1894 ptr:    71 data: 0xC1DB8285 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33413  ts: 0x157C835E76D13DDE
 ***            ADC data: block:  1894 ptr:    73 data: 0xC1DD7FA3 module:  7 fail: 0 range: 0 id:   477 channel: 29 adc: 32675  ts: 0x157C835E76D145AE
 ***            ADC data: block:  1894 ptr:    75 data: 0xC1EC83C1 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33729  ts: 0x157C835E76D16CBE
 ***            ADC data: block:  1894 ptr:    77 data: 0xC1F181BE module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33214  ts: 0x157C835E76D18BFE
 ***            ADC data: block:  1894 ptr:    79 data: 0xC1E980C6 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32966  ts: 0x157C835E76D193CE
 ***            ADC data: block:  1894 ptr:    81 data: 0xC1F48267 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33383  ts: 0x157C835E76D193CE
 ***            ADC data: block:  1894 ptr:    83 data: 0xC1D0807C module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32892  ts: 0x157C835E76D1AB3E
 ***            ADC data: block:  1894 ptr:    85 data: 0xC1E181E3 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33251  ts: 0x157C835E76D1AB3E
 ***            ADC data: block:  1894 ptr:    87 data: 0xC1CB816A module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33130  ts: 0x157C835E76D1B30E
 ***            ADC data: block:  1894 ptr:    89 data: 0xC1D68227 module:  7 fail: 0 range: 0 id:   470 channel: 22 adc: 33319  ts: 0x157C835E76D1B30E
 ***            ADC data: block:  1894 ptr:    91 data: 0xC1E381D6 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33238  ts: 0x157C835E76D1B30E
 ***            ADC data: block:  1894 ptr:    93 data: 0xC1CD8221 module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33313  ts: 0x157C835E76D1BADE
 ***            ADC data: block:  1894 ptr:    95 data: 0xC1D7805B module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32859  ts: 0x157C835E76D1CA7E
 ***            ADC data: block:  1894 ptr:    97 data: 0xC1C580F3 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33011  ts: 0x157C835E76D1D24E
 ***            ADC data: block:  1894 ptr:    99 data: 0xC1DB8190 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33168  ts: 0x157C835E76D1D24E
 ***            ADC data: block:  1894 ptr:   101 data: 0xC1FD8167 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33127  ts: 0x157C835E76D1E9BE
 ***            ADC data: block:  1894 ptr:   103 data: 0xC1C18102 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33026  ts: 0x157C835E76D1F95E
 ***            ADC data: block:  1894 ptr:   105 data: 0xC1E0811D module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33053  ts: 0x157C835E76D1F95E
 ***            ADC data: block:  1894 ptr:   107 data: 0xC1EC83F9 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33785  ts: 0x157C835E76D2012E
 ***            ADC data: block:  1894 ptr:   109 data: 0xC1F1821B module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33307  ts: 0x157C835E76D2206E
 ***            ADC data: block:  1894 ptr:   111 data: 0xC1CB819F module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 33183  ts: 0x157C835E76D23FAE
/ptr: 13001
...skipping
 ***            ADC data: block:  1894 ptr: 13001 data: 0xC1F78041 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32833  ts: 0x157C835E79FA566E
 ***            ADC data: block:  1894 ptr: 13003 data: 0xC1FF8139 module:  7 fail: 0 range: 0 id:   511 channel: 63 adc: 33081  ts: 0x157C835E79FA5E3E
 ***            ADC data: block:  1894 ptr: 13005 data: 0xC1D08064 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32868  ts: 0x157C835E79FAAC5E
 ***            ADC data: block:  1894 ptr: 13007 data: 0xC1D0806C module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32876  ts: 0x157C835E79FB40CE
 ***            ADC data: block:  1894 ptr: 13009 data: 0xC1D0807E module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32894  ts: 0x157C835E79FBD53E
 ***            ADC data: block:  1894 ptr: 13011 data: 0xC1E08247 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33351  ts: 0x157C835E79FCBF9E
 ***            ADC data: block:  1894 ptr: 13013 data: 0xC1D280D5 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32981  ts: 0x157C835E79FCC76E
 ***            ADC data: block:  1894 ptr: 13015 data: 0xC1F18345 module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33605  ts: 0x157C835E79FCC76E
 ***            ADC data: block:  1894 ptr: 13017 data: 0xC1F181DE module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33246  ts: 0x157C835E79FD5BDE
 ***            ADC data: block:  1894 ptr: 13019 data: 0xC1F1819D module:  7 fail: 0 range: 0 id:   497 channel: 49 adc: 33181  ts: 0x157C835E79FDF04E
 ***            ADC data: block:  1894 ptr: 13021 data: 0xC1E080A8 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 32936  ts: 0x157C835E79FDFFEE
 ***            ADC data: block:  1894 ptr: 13023 data: 0xC1E080DE module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 32990  ts: 0x157C835E79FF01BE
 ***            ADC data: block:  1894 ptr: 13025 data: 0xC1EF8124 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33060  ts: 0x157C835E79FF28CE
 ***            ADC data: block:  1894 ptr: 13027 data: 0xC1C581F2 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33266  ts: 0x157C835E79FF309E
 ***            ADC data: block:  1894 ptr: 13029 data: 0xC1F281AF module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33199  ts: 0x157C835E79FF309E
 ***            ADC data: block:  1894 ptr: 13031 data: 0xC1E080CA module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 32970  ts: 0x157C835E79FF8E5E
 ***            ADC data: block:  1894 ptr: 13033 data: 0xC1EF8103 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33027  ts: 0x157C835E79FFB56E
 ***            ADC data: block:  1894 ptr: 13035 data: 0xC1C58122 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33058  ts: 0x157C835E79FFC50E
 ***            ADC data: block:  1894 ptr: 13037 data: 0xC1F2817D module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33149  ts: 0x157C835E79FFC50E
 ***   WR48-64 timestamp: block:  1894 ptr: 13039 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157C835E7A000000
 ***   SYNC100 timestamp: block:  1894 ptr: 13041 data: 0x874835E5 module:  7 information type:  4 information field: 0x000835E5 ts: 0x157C835E5A000000 OLD: 
ts: 0x157C835E7A000000 dt: 0x0000000020000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1894 ptr: 13041 data: 0x874835E5 module:  7 information type:  4 information field: 0x000835E5 ts: 0x157C835E5A000000
 ***            ADC data: block:  1894 ptr: 13043 data: 0xC1F281D2 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33234  ts: 0x157C835E5A00691E
 ***            ADC data: block:  1894 ptr: 13045 data: 0xC1EF80BD module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32957  ts: 0x157C835E5A00B73E
 ***            ADC data: block:  1894 ptr: 13047 data: 0xC1F27FC4 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32708  ts: 0x157C835E5A00FD8E
 ***            ADC data: block:  1894 ptr: 13049 data: 0xC1E08107 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33031  ts: 0x157C835E5A012C6E
 ***            ADC data: block:  1894 ptr: 13051 data: 0xC1EF80CC module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32972  ts: 0x157C835E5A0143DE
 ***            ADC data: block:  1894 ptr: 13053 data: 0xC1F28107 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33031  ts: 0x157C835E5A0191FE
 ***            ADC data: block:  1894 ptr: 13055 data: 0xC1E0812C module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33068  ts: 0x157C835E5A01B90E
 ***            ADC data: block:  1894 ptr: 13057 data: 0xC1EF80F4 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33012  ts: 0x157C835E5A01FF5E
 ***            ADC data: block:  1894 ptr: 13059 data: 0xC1F2822B module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33323  ts: 0x157C835E5A02266E
 ***            ADC data: block:  1894 ptr: 13061 data: 0xC1E0820D module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 33293  ts: 0x157C835E5A0245AE
 ***            ADC data: block:  1894 ptr: 13063 data: 0xC1EF80C7 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32967  ts: 0x157C835E5A028BFE
 ***            ADC data: block:  1894 ptr: 13065 data: 0xC1F28123 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33059  ts: 0x157C835E5A02BADE
 ***            ADC data: block:  1894 ptr: 13067 data: 0xC1E08089 module:  7 fail: 0 range: 0 id:   480 channel: 32 adc: 32905  ts: 0x157C835E5A02D24E
 ***            ADC data: block:  1894 ptr: 13069 data: 0xC1EF805F module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32863  ts: 0x157C835E5A03189E
 ***            ADC data: block:  1894 ptr: 13071 data: 0xC1EF80EB module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33003  ts: 0x157C835E5A03EB8E
 ***            ADC data: block:  1894 ptr: 13073 data: 0xC1EF80E5 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 32997  ts: 0x157C835E5A047FFE
 ***            ADC data: block:  1894 ptr: 13075 data: 0xC1C581B5 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33205  ts: 0x157C835E5A04ED5E
 ***            ADC data: block:  1894 ptr: 13077 data: 0xC1E181EC module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33260  ts: 0x157C835E5A0C676E
 ***            ADC data: block:  1894 ptr: 13079 data: 0xC1F481AD module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33197  ts: 0x157C835E5A0C676E
 ***            ADC data: block:  1894 ptr: 13081 data: 0xC1E18010 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32784  ts: 0x157C835E5A0CFBDE
 ***            ADC data: block:  1894 ptr: 13083 data: 0xC1F4805F module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32863  ts: 0x157C835E5A0CFBDE
 ***            ADC data: block:  1894 ptr: 13085 data: 0xC1E180D8 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32984  ts: 0x157C835E5A0D904E
 ***            ADC data: block:  1894 ptr: 13087 data: 0xC1F48134 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33076  ts: 0x157C835E5A0D904E
 ***            ADC data: block:  1894 ptr: 13089 data: 0xC1E1807E module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32894  ts: 0x157C835E5A0E24BE
 ***            ADC data: block:  1894 ptr: 13091 data: 0xC1F480C8 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32968  ts: 0x157C835E5A0E24BE
 ***            ADC data: block:  1894 ptr: 13093 data: 0xC1C580A2 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32930  ts: 0x157C835E5A0ED86E
 ***            ADC data: block:  1894 ptr: 13095 data: 0xC1FD81D1 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33233  ts: 0x157C835E5A0ED86E
 ***            ADC data: block:  1894 ptr: 13097 data: 0xC1C580DD module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 32989  ts: 0x157C835E5A0F6CDE
 ***            ADC data: block:  1894 ptr: 13099 data: 0xC1FD8142 module:  7 fail: 0 range: 0 id:   509 channel: 61 adc: 33090  ts: 0x157C835E5A0F6CDE
 ***            ADC data: block:  1894 ptr: 13101 data: 0xC1C58189 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33161  ts: 0x157C835E5A1010EE

/1895 ptr:  7801
...skipping
 ***            ADC data: block:  1895 ptr:  7801 data: 0xC1C180E4 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 32996  ts: 0x157C835E5C3DCE2E
 ***            ADC data: block:  1895 ptr:  7803 data: 0xC1E180CC module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32972  ts: 0x157C835E5C3DCE2E
 ***            ADC data: block:  1895 ptr:  7805 data: 0xC1C87FC3 module:  7 fail: 0 range: 0 id:   456 channel:  8 adc: 32707  ts: 0x157C835E5C3DD5FE
 ***            ADC data: block:  1895 ptr:  7807 data: 0xC1D28094 module:  7 fail: 0 range: 0 id:   466 channel: 18 adc: 32916  ts: 0x157C835E5C3DD5FE
 ***            ADC data: block:  1895 ptr:  7809 data: 0xC1E38117 module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33047  ts: 0x157C835E5C3DD5FE
 ***            ADC data: block:  1895 ptr:  7811 data: 0xC1F3800A module:  7 fail: 0 range: 0 id:   499 channel: 51 adc: 32778  ts: 0x157C835E5C3DD5FE
 ***            ADC data: block:  1895 ptr:  7813 data: 0xC1CB80D3 module:  7 fail: 0 range: 0 id:   459 channel: 11 adc: 32979  ts: 0x157C835E5C3DDDCE
 ***            ADC data: block:  1895 ptr:  7815 data: 0xC1D4812D module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33069  ts: 0x157C835E5C3DDDCE
 ***            ADC data: block:  1895 ptr:  7817 data: 0xC1E68062 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32866  ts: 0x157C835E5C3DDDCE
 ***            ADC data: block:  1895 ptr:  7819 data: 0xC1F4812B module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33067  ts: 0x157C835E5C3DDDCE
 ***            ADC data: block:  1895 ptr:  7821 data: 0xC1CC8030 module:  7 fail: 0 range: 0 id:   460 channel: 12 adc: 32816  ts: 0x157C835E5C3DE59E
 ***            ADC data: block:  1895 ptr:  7823 data: 0xC1D78019 module:  7 fail: 0 range: 0 id:   471 channel: 23 adc: 32793  ts: 0x157C835E5C3DE59E
 ***            ADC data: block:  1895 ptr:  7825 data: 0xC1E97FC0 module:  7 fail: 0 range: 0 id:   489 channel: 41 adc: 32704  ts: 0x157C835E5C3DE59E
 ***            ADC data: block:  1895 ptr:  7827 data: 0xC1FC80C2 module:  7 fail: 0 range: 0 id:   508 channel: 60 adc: 32962  ts: 0x157C835E5C3DE59E
 ***            ADC data: block:  1895 ptr:  7829 data: 0xC1CD814A module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33098  ts: 0x157C835E5C3DED6E
 ***            ADC data: block:  1895 ptr:  7831 data: 0xC1DB81FE module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33278  ts: 0x157C835E5C3DED6E
 ***            ADC data: block:  1895 ptr:  7833 data: 0xC1EB80FF module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 33023  ts: 0x157C835E5C3DED6E
 ***            ADC data: block:  1895 ptr:  7835 data: 0xC1DC805A module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32858  ts: 0x157C835E5C3DF53E
 ***            ADC data: block:  1895 ptr:  7837 data: 0xC1EC81B6 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33206  ts: 0x157C835E5C3DF53E
 ***            ADC data: block:  1895 ptr:  7839 data: 0xC1EF80E9 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33001  ts: 0x157C835E5C3DFD0E
 ***            ADC data: block:  1895 ptr:  7841 data: 0xC1C3811E module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33054  ts: 0x157C835E5C3E435E
 ***            ADC data: block:  1895 ptr:  7843 data: 0xC1E17F83 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 32643  ts: 0x157C835E5C3E629E
 ***            ADC data: block:  1895 ptr:  7845 data: 0xC1F481E3 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33251  ts: 0x157C835E5C3E6A6E
 ***            ADC data: block:  1895 ptr:  7847 data: 0xC1C1804E module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 32846  ts: 0x157C835E5C3E89AE
 ***            ADC data: block:  1895 ptr:  7849 data: 0xC1D481DB module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33243  ts: 0x157C835E5C3E89AE
 ***            ADC data: block:  1895 ptr:  7851 data: 0xC1E68110 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 33040  ts: 0x157C835E5C3E89AE
 ***            ADC data: block:  1895 ptr:  7853 data: 0xC1DC7FAB module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32683  ts: 0x157C835E5C3E917E
 ***            ADC data: block:  1895 ptr:  7855 data: 0xC1EC80E4 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 32996  ts: 0x157C835E5C3E917E
 ***            ADC data: block:  1895 ptr:  7857 data: 0xC1EF80F4 module:  7 fail: 0 range: 0 id:   495 channel: 47 adc: 33012  ts: 0x157C835E5C3E994E
 ***            ADC data: block:  1895 ptr:  7859 data: 0xC1DB8160 module:  7 fail: 0 range: 0 id:   475 channel: 27 adc: 33120  ts: 0x157C835E5C3EC82E
 ***            ADC data: block:  1895 ptr:  7861 data: 0xC1C381AC module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33196  ts: 0x157C835E5C3ECFFE
 ***            ADC data: block:  1895 ptr:  7863 data: 0xC1E3812F module:  7 fail: 0 range: 0 id:   483 channel: 35 adc: 33071  ts: 0x157C835E5C3ECFFE
 ***            ADC data: block:  1895 ptr:  7865 data: 0xC1CD811A module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33050  ts: 0x157C835E5C3ED7CE
 ***            ADC data: block:  1895 ptr:  7867 data: 0xC1EB80DE module:  7 fail: 0 range: 0 id:   491 channel: 43 adc: 32990  ts: 0x157C835E5C3ED7CE
 ***            ADC data: block:  1895 ptr:  7869 data: 0xC1E18133 module:  7 fail: 0 range: 0 id:   481 channel: 33 adc: 33075  ts: 0x157C835E5C3EEF3E
 ***            ADC data: block:  1895 ptr:  7871 data: 0xC1F481F0 module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33264  ts: 0x157C835E5C3EFEDE
 ***            ADC data: block:  1895 ptr:  7873 data: 0xC1E680F9 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 33017  ts: 0x157C835E5C3F164E
 ***            ADC data: block:  1895 ptr:  7875 data: 0xC1C1818B module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33163  ts: 0x157C835E5C3F1E1E
 ***            ADC data: block:  1895 ptr:  7877 data: 0xC1D481E7 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33255  ts: 0x157C835E5C3F1E1E
 ***            ADC data: block:  1895 ptr:  7879 data: 0xC1D48169 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33129  ts: 0x157C835E58849AFE OLD: 
ts: 0x157C835E5C3F1E1E dt: 0x0000000003BA8320 tag: 3 type: 0
 ***            ADC data: block:  1895 ptr:  7879 data: 0xC1D48169 module:  7 fail: 0 range: 0 id:   468 channel: 20 adc: 33129  ts: 0x157C835E58849AFE
 ***            ADC data: block:  1895 ptr:  7881 data: 0xC1C680C9 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 32969  ts: 0x157C835E5884A2CE
 ***            ADC data: block:  1895 ptr:  7883 data: 0xC1C18217 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33303  ts: 0x157C835E588546DE
 ***            ADC data: block:  1895 ptr:  7885 data: 0xC1C68142 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33090  ts: 0x157C835E58854EAE
 ***            ADC data: block:  1895 ptr:  7887 data: 0xC1C18126 module:  7 fail: 0 range: 0 id:   449 channel:  1 adc: 33062  ts: 0x157C835E5885D37E
 ***            ADC data: block:  1895 ptr:  7889 data: 0xC1C680F8 module:  7 fail: 0 range: 0 id:   454 channel:  6 adc: 33016  ts: 0x157C835E5885DB4E
 ***            ADC data: block:  1895 ptr:  7891 data: 0xC1C0815D module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33117  ts: 0x157C835E5886219E
 ***            ADC data: block:  1895 ptr:  7893 data: 0xC1F780C6 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32966  ts: 0x157C835E5886219E
 ***            ADC data: block:  1895 ptr:  7895 data: 0xC1CC802F module:  7 fail: 0 range: 0 id:   460 channel: 12 adc: 32815  ts: 0x157C835E5886296E
 ***            ADC data: block:  1895 ptr:  7897 data: 0xC1C081CF module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 33231  ts: 0x157C835E5886AE3E
 ***            ADC data: block:  1895 ptr:  7899 data: 0xC1CC80C2 module:  7 fail: 0 range: 0 id:   460 channel: 12 adc: 32962  ts: 0x157C835E5886B60E
 ***            ADC data: block:  1895 ptr:  7901 data: 0xC1F78198 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33176  ts: 0x157C835E5886B60E
 ***            ADC data: block:  1895 ptr:  7903 data: 0xC1C080AD module:  7 fail: 0 range: 0 id:   448 channel:  0 adc: 32941  ts: 0x157C835E58873ADE

/1943 ptr:  5089
...skipping
 ***            ADC data: block:  1943 ptr:  5089 data: 0xC1E680B2 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32946  ts: 0x157C835F5FFEC26E
 ***            ADC data: block:  1943 ptr:  5091 data: 0xC1D07FD1 module:  7 fail: 0 range: 0 id:   464 channel: 16 adc: 32721  ts: 0x157C835F5FFED9DE
 ***            ADC data: block:  1943 ptr:  5093 data: 0xC1DC80A7 module:  7 fail: 0 range: 0 id:   476 channel: 28 adc: 32935  ts: 0x157C835F5FFEE1AE
 ***            ADC data: block:  1943 ptr:  5095 data: 0xC1EC816D module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33133  ts: 0x157C835F5FFF08BE
 ***            ADC data: block:  1943 ptr:  5097 data: 0xC1E681B1 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 33201  ts: 0x157C835F5FFF4F0E
 ***            ADC data: block:  1943 ptr:  5099 data: 0xC1C581D8 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33240  ts: 0x157C835F5FFF6E4E
 ***            ADC data: block:  1943 ptr:  5101 data: 0xC1F281C9 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33225  ts: 0x157C835F5FFF6E4E
 ***            ADC data: block:  1943 ptr:  5103 data: 0xC1F4812F module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 33071  ts: 0x157C835F5FFF761E
 ***            ADC data: block:  1943 ptr:  5105 data: 0xC1EC8353 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33619  ts: 0x157C835F5FFF955E
 ***            ADC data: block:  1943 ptr:  5107 data: 0xC1E68038 module:  7 fail: 0 range: 0 id:   486 channel: 38 adc: 32824  ts: 0x157C835F5FFFDBAE
 ***            ADC data: block:  1943 ptr:  5109 data: 0xC1F281FF module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 33279  ts: 0x157C835F5FFFFAEE
 ***   WR48-64 timestamp: block:  1943 ptr:  5111 data: 0x8750157C module:  7 information type:  5 information field: 0x0000157C ts: 0x157C835F500002BE
 ***   SYNC100 timestamp: block:  1943 ptr:  5113 data: 0x874835E8 module:  7 information type:  4 information field: 0x000835E8 ts: 0x157C835E800002BE OLD: 
ts: 0x157C835F500002BE dt: 0x00000000D0000000 tag: 2 type: 5
 ***   SYNC100 timestamp: block:  1943 ptr:  5113 data: 0x874835E8 module:  7 information type:  4 information field: 0x000835E8 ts: 0x157C835E800002BE
 ***            ADC data: block:  1943 ptr:  5115 data: 0xC1C58185 module:  7 fail: 0 range: 0 id:   453 channel:  5 adc: 33157  ts: 0x157C835E800002BE
 ***            ADC data: block:  1943 ptr:  5117 data: 0xC1F480BA module:  7 fail: 0 range: 0 id:   500 channel: 52 adc: 32954  ts: 0x157C835E800002BE
 ***            ADC data: block:  1943 ptr:  5119 data: 0xC1EC8299 module:  7 fail: 0 range: 0 id:   492 channel: 44 adc: 33433  ts: 0x157C835E800021FE
 ***            ADC data: block:  1943 ptr:  5121 data: 0xC1C38154 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 33108  ts: 0x157C835E8000490E
... 209 more lines ...
Attachment 3: analyser.f
      PROGRAM GREAT

C     Read and analyse GREAT format data - version 3.3.0
C
C     See:
C     http://npg.dl.ac.uk/documents/edoc504/edoc504.html
C     http://npg.dl.ac.uk/DataAcq/TSformat.html

      IMPLICIT NONE                                 
C
C     External functions
C
      EXTERNAL timestamp
C
      DOUBLE PRECISION dfloat
C
      INTEGER and
      INTEGER*8 iargc
      INTEGER rshift
      INTEGER*8 timestamp
C
      REAL secnds
C
C     Parameter variables
C
      INTEGER block_length
      PARAMETER (block_length = 16384)
C
C     Local variables
C
      CHARACTER*255 arg
      CHARACTER*(block_length*4) buffer
      CHARACTER*255 filename
C
      INTEGER adc_data, block_number, block_ptr, ch, channel
      INTEGER channel_ident, data_block(block_length)
      INTEGER*8 dead(32)
      INTEGER event, fail
      INTEGER*8 first_ts
      INTEGER i
      INTEGER*2 i2(2), i2_2(4)
      INTEGER i4, i4_2(2)
      INTEGER*8 i8, idle(32)
      INTEGER information, information_index, io_error, items(0:32,0:31)
      INTEGER j, j4
      INTEGER*8 last_ts
      INTEGER mbs_data, module
      INTEGER*8 pause(32)
      INTEGER range
      INTEGER*8 resume(32)
      INTEGER sample_length, sum(0:31)
      INTEGER*8 sync(32)
      INTEGER tag, tag_old
      INTEGER*8 ts
      INTEGER ts28, ts48, ts64
      INTEGER*8 ts_old
      INTEGER type, type_old
C
      LOGICAL verbose
C
      REAL dt, t1, t2

      EQUIVALENCE ( data_block, buffer )
      EQUIVALENCE ( i4, i2( 1 ) )
      EQUIVALENCE ( i8, i4_2(1) )

      SAVE data_block, block_ptr, block_number, event

C----67---------------------------------------------------------------72------80
C     Announce program

      WRITE( 6, 9000 )

C     Initialise variables

      block_ptr = 1
      block_number = 0
      event = 0

      i4 = 0
      i8 = 0

      ts_old = 0
      tag_old = 0
      type_old = 0
      first_ts = 0
      last_ts = 0
      ts28 = 0
      ts48 = 0
      ts64 = 0

      type = 0
      
      verbose = .FALSE.
      
      DO i = 0, 32
       DO j = 0, 31
        sum( j ) = 0
        items( i, j ) = 0
       ENDDO
      ENDDO

      DO i = 1, 32
       pause( i ) = -1
       resume( i ) = -1
       dead( i ) = 0
       sync( i ) = 0
       idle( i ) = 0
      ENDDO

C     Get input filename from command line argument

      DO i = 1, iargc()
       CALL getarg( i, arg )
       IF ( arg.EQ.'v') THEN
        verbose = .TRUE.
        write(6,*) ' verbose'
       ELSE
        filename = arg
       ENDIF
      ENDDO
      IF ( iargc().LT.1 .OR. iargc().GT.2 ) THEN
       WRITE( 6, * ) ' Usage: ./a.exe [v] <filename>'
       STOP
      ENDIF

      t1 = SECNDS( 0.0 )

C     Open file

      OPEN( 10, RECL = 4 * block_length,
     +      FILE = filename, STATUS = 'OLD', ACCESS = 'DIRECT' )

    1 CONTINUE

C     Read next block
C     Note: for FORTRAN direct I/O first block/record number is 1
C           for GREAT data format first (encoded) block/record number is 0

      block_number = block_number + 1

      READ( 10, REC = block_number, iostat=io_error, ERR=1000 ) buffer

      DO block_ptr = 7, data_block( 6 ) / 4 + 4, 2


       i4 = data_block( block_ptr )
       j4 = data_block( block_ptr + 1 )

       tag = AND( RSHIFT( i4, 30 ), Z'00000003' )

       module = 0
       type = 0

C      tag = 3 = ADC data format
C      tag = 2 = Other data
C      tag = 1 = Sample trace buffer format
C      tag = 0 = undefined?

C----67---------------------------------------------------------------72------80
C     ADC data

       IF ( tag.EQ.3 ) THEN

        fail = AND( RSHIFT( i4, 29 ), Z'00000001' )

        IF ( fail.NE.0 ) THEN

         WRITE( 6, 9600 ) block_number, block_ptr, i4

        ENDIF

        range = AND( RSHIFT( i4, 28 ), Z'00000001' )
        channel_ident = AND( RSHIFT( i4, 16 ), Z'00000fff' )
        module = AND( RSHIFT( channel_ident, 6 ), Z'0000003f' )
        IF ( range.EQ.1 ) THEN
         items(module, 19 ) = items(module, 19 ) + 1
        ENDIF
        channel = AND( channel_ident, Z'0000003f' )
        adc_data = AND( i4, Z'0000ffff' )

        items(module, 0 ) = items(module, 0 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        ts = timestamp( ts64, ts48, ts28, type )

	IF ( ts.LT.ts_old ) THEN

	 items(module, 20 ) = items(module, 20 ) + 1

         WRITE( 6, 9710 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts, ts_old, ts_old-ts,
     +   tag_old, type_old
        ENDIF

        ts_old = ts
        tag_old = tag
        type_old = 0

        IF ( verbose ) THEN 
         WRITE( 6, 9700 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts
        ENDIF

        ch = channel + ( module - 1 ) * 64 + ( range * 2048 )

C----67---------------------------------------------------------------72------80
C     Other information

       ELSEIF( tag.EQ.2 ) THEN

        module = AND( RSHIFT( i4, 24 ), Z'0000003f' )
        type = AND( RSHIFT( i4, 20 ), Z'0000000f' )
        information = AND( i4, Z'000fffff' )

        items(module, 1 ) = items(module, 1 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        IF ( type.EQ.2 ) THEN

C----67---------------------------------------------------------------72------80
C     PAUSE timestamp

         items(module, 4 ) = items(module, 4 ) + 1

        ts48 = information
C        ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 21 ) = items(module, 21 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         pause( module ) = ts

         IF ( verbose ) THEN 
          WRITE( 6, 9800 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.3 ) THEN

C----67---------------------------------------------------------------72------80
C     RESUME timestamp

         items(module, 5 ) = items(module, 5 ) + 1

         ts48 = information
C         ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 22 ) = items(module, 22 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         resume( module ) = ts

         IF ( resume( module ).GE.0 
     +         .AND.
     +        pause( module ).GE.0 
     +          .AND.
     +        resume( module ).GE.pause( module ) ) THEN
          dead( module ) = ( resume( module ) - pause( module ) ) 
     +                     + dead( module )
          pause( module ) = -1
          resume( module ) = -1
         ENDIF

         IF ( verbose ) THEN 
          WRITE( 6, 9810 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.4 ) THEN

C----67---------------------------------------------------------------72------80
C     SYNC100 timestamp/White Rabbit Timestamp Marker (bits 28-47)

         items(module, 6 ) = items(module, 6 ) + 1

         ts48 = information

C        ts = timestamp( ts48, ts28, type )
         ts = timestamp( ts64, ts48, ts28, type )


         IF ( first_ts.EQ.0 ) THEN
          first_ts = ts 
... 451 more lines ...
Attachment 4: analyser.f
      PROGRAM GREAT

C     Read and analyse GREAT format data - version 3.3.0
C
C     See:
C     http://npg.dl.ac.uk/documents/edoc504/edoc504.html
C     http://npg.dl.ac.uk/DataAcq/TSformat.html

      IMPLICIT NONE                                 
C
C     External functions
C
      EXTERNAL timestamp
C
      DOUBLE PRECISION dfloat
C
      INTEGER and
      INTEGER*8 iargc
      INTEGER rshift
      INTEGER*8 timestamp
C
      REAL secnds
C
C     Parameter variables
C
      INTEGER block_length
      PARAMETER (block_length = 16384)
C
C     Local variables
C
      CHARACTER*255 arg
      CHARACTER*(block_length*4) buffer
      CHARACTER*255 filename
C
      INTEGER adc_data, block_number, block_ptr, ch, channel
      INTEGER channel_ident, data_block(block_length)
      INTEGER*8 dead(32)
      INTEGER event, fail
      INTEGER*8 first_ts
      INTEGER i
      INTEGER*2 i2(2), i2_2(4)
      INTEGER i4, i4_2(2)
      INTEGER*8 i8, idle(32)
      INTEGER information, information_index, io_error, items(0:32,0:31)
      INTEGER j, j4
      INTEGER*8 last_ts
      INTEGER mbs_data, module
      INTEGER*8 pause(32)
      INTEGER range
      INTEGER*8 resume(32)
      INTEGER sample_length, sum(0:31)
      INTEGER*8 sync(32)
      INTEGER tag, tag_old
      INTEGER*8 ts
      INTEGER ts28, ts48, ts64
      INTEGER*8 ts_old
      INTEGER type, type_old
C
      LOGICAL verbose
C
      REAL dt, t1, t2

      EQUIVALENCE ( data_block, buffer )
      EQUIVALENCE ( i4, i2( 1 ) )
      EQUIVALENCE ( i8, i4_2(1) )

      SAVE data_block, block_ptr, block_number, event

C----67---------------------------------------------------------------72------80
C     Announce program

      WRITE( 6, 9000 )

C     Initialise variables

      block_ptr = 1
      block_number = 0
      event = 0

      i4 = 0
      i8 = 0

      ts_old = 0
      tag_old = 0
      type_old = 0
      first_ts = 0
      last_ts = 0
      ts28 = 0
      ts48 = 0
      ts64 = 0

      type = 0
      
      verbose = .FALSE.
      
      DO i = 0, 32
       DO j = 0, 31
        sum( j ) = 0
        items( i, j ) = 0
       ENDDO
      ENDDO

      DO i = 1, 32
       pause( i ) = -1
       resume( i ) = -1
       dead( i ) = 0
       sync( i ) = 0
       idle( i ) = 0
      ENDDO

C     Get input filename from command line argument

      DO i = 1, iargc()
       CALL getarg( i, arg )
       IF ( arg.EQ.'v') THEN
        verbose = .TRUE.
        write(6,*) ' verbose'
       ELSE
        filename = arg
       ENDIF
      ENDDO
      IF ( iargc().LT.1 .OR. iargc().GT.2 ) THEN
       WRITE( 6, * ) ' Usage: ./a.exe [v] <filename>'
       STOP
      ENDIF

      t1 = SECNDS( 0.0 )

C     Open file

      OPEN( 10, RECL = 4 * block_length,
     +      FILE = filename, STATUS = 'OLD', ACCESS = 'DIRECT' )

    1 CONTINUE

C     Read next block
C     Note: for FORTRAN direct I/O first block/record number is 1
C           for GREAT data format first (encoded) block/record number is 0

      block_number = block_number + 1

      READ( 10, REC = block_number, iostat=io_error, ERR=1000 ) buffer

      DO block_ptr = 7, data_block( 6 ) / 4 + 4, 2


       i4 = data_block( block_ptr )
       j4 = data_block( block_ptr + 1 )

       tag = AND( RSHIFT( i4, 30 ), Z'00000003' )

       module = 0
       type = 0

C      tag = 3 = ADC data format
C      tag = 2 = Other data
C      tag = 1 = Sample trace buffer format
C      tag = 0 = undefined?

C----67---------------------------------------------------------------72------80
C     ADC data

       IF ( tag.EQ.3 ) THEN

        fail = AND( RSHIFT( i4, 29 ), Z'00000001' )

        IF ( fail.NE.0 ) THEN

         WRITE( 6, 9600 ) block_number, block_ptr, i4

        ENDIF

        range = AND( RSHIFT( i4, 28 ), Z'00000001' )
        channel_ident = AND( RSHIFT( i4, 16 ), Z'00000fff' )
        module = AND( RSHIFT( channel_ident, 6 ), Z'0000003f' )
        IF ( range.EQ.1 ) THEN
         items(module, 19 ) = items(module, 19 ) + 1
        ENDIF
        channel = AND( channel_ident, Z'0000003f' )
        adc_data = AND( i4, Z'0000ffff' )

        items(module, 0 ) = items(module, 0 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        ts = timestamp( ts64, ts48, ts28, type )

	IF ( ts.LT.ts_old ) THEN

	 items(module, 20 ) = items(module, 20 ) + 1

         WRITE( 6, 9710 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts, ts_old, ts_old-ts,
     +   tag_old, type_old
        ENDIF

        ts_old = ts
        tag_old = tag
        type_old = 0

        IF ( verbose ) THEN 
         WRITE( 6, 9700 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts
        ENDIF

        ch = channel + ( module - 1 ) * 64 + ( range * 2048 )

C----67---------------------------------------------------------------72------80
C     Other information

       ELSEIF( tag.EQ.2 ) THEN

        module = AND( RSHIFT( i4, 24 ), Z'0000003f' )
        type = AND( RSHIFT( i4, 20 ), Z'0000000f' )
        information = AND( i4, Z'000fffff' )

        items(module, 1 ) = items(module, 1 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        IF ( type.EQ.2 ) THEN

C----67---------------------------------------------------------------72------80
C     PAUSE timestamp

         items(module, 4 ) = items(module, 4 ) + 1

        ts48 = information
C        ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 21 ) = items(module, 21 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         pause( module ) = ts

         IF ( verbose ) THEN 
          WRITE( 6, 9800 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.3 ) THEN

C----67---------------------------------------------------------------72------80
C     RESUME timestamp

         items(module, 5 ) = items(module, 5 ) + 1

         ts48 = information
C         ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 22 ) = items(module, 22 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         resume( module ) = ts

         IF ( resume( module ).GE.0 
     +         .AND.
     +        pause( module ).GE.0 
     +          .AND.
     +        resume( module ).GE.pause( module ) ) THEN
          dead( module ) = ( resume( module ) - pause( module ) ) 
     +                     + dead( module )
          pause( module ) = -1
          resume( module ) = -1
         ENDIF

         IF ( verbose ) THEN 
          WRITE( 6, 9810 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.4 ) THEN

C----67---------------------------------------------------------------72------80
C     SYNC100 timestamp/White Rabbit Timestamp Marker (bits 28-47)

         items(module, 6 ) = items(module, 6 ) + 1

         ts48 = information

C        ts = timestamp( ts48, ts28, type )
         ts = timestamp( ts64, ts48, ts28, type )


         IF ( first_ts.EQ.0 ) THEN
          first_ts = ts 
... 446 more lines ...
  19   Thu Jan 24 10:26:09 2019 VP, TD, CA, NHStart up Merger
1) Click on item "Merger MK2" in top panel
   This kills any existing Merger existence and loads a new copy.

2) Start (if needed) browser conection to localhost:8115.

3) Click on "NewMerger Control"

4) In NewMerger Control, click "Reload"

5) If MergerState = IDLE, click "SETUP".

6) Wait until state = STOPed

7) If links shown are correct, click "GO"

8) Wait until popup about "Resume Merger" appears and click OK

9) when writing to disk, click "Forward Output to Data"
Attachment 1: merger_screenshot.png
merger_screenshot.png
  18   Thu Jan 24 10:21:26 2019 CA, TD, NH, VPJanuary 24th 2019
10.30 AIDA setup and DAQ start complete
      writing to file TapeData/NULL/R1

      FEE64 Temperatures and Good event statistics ok (attachments 1 and 2)

11.00 results of analyser - shows significant number of timewarps in aida07, and a few in aida01-04

11.24 compared timestamps between mbs and AIDA, agree to 8 digits (attachment 3)

12.49 DAQ and Merger stopped and restarted

      Raised slow comparator threshold from 0xa to 0x64

      Raised fast comparator threshold (LEC/MEC) from 0xf to 0xff

12.51 Good event statistics: aida08 running at a normal rate

      pulser peak (1.8.L) no longer has noise shoulder (attachment 4)

      writing to file TapeData/NULL/R2


13.58 stopped writing to file. Pulser frequency increased to 1 kHz

      writing to file TapeData/NULL/R3

15.00 file R4 - DAQ startup

17.58 trimmed and re-crimped ground wire on aida08
      lowered pulser freq back to 50Hz
      reduced slow comparator threshold to 0xa, fast comparator threshold (LEC) to 0xf

      good event stats ok, aida08 no longer running hot (attachment 6)
      1.8.L and 1.8.W spectra of aida08 (attachment 5)
      pulser peak width now at 55.71

      FEE64 temperatures ok (attachment 7)
Attachment 1: 240119_stats.png
240119_stats.png
Attachment 2: 240119_temp.png
240119_temp.png
Attachment 3: 240119_timestamps.png
240119_timestamps.png
Attachment 4: 240119_fee8_18L.png
240119_fee8_18L.png
Attachment 5: 240119_spectra.png
240119_spectra.png
Attachment 6: 1740Statsafter240119.png
1740Statsafter240119.png
Attachment 7: 240119_tempafter.png
240119_tempafter.png
  17   Wed Jan 23 17:03:08 2019 CA, TD, NH, VPJanuary 23rd 2019
Post White Rabbit update

Attachment 1 - 1.8.W pulser waveform spectra for all FEE64
             - pulser signals observed in all FEE64

Attachment 2 - good event statistics - aida08 has significant noise

Attachment 3 - FEE64 temperatures - all ok, but firmware version not displayed

Attachment 4 - 1.8.L pulser peak spectra

             - widths:

             aida01 - 23.66
             aida02 - 19.76
             aida03 - 20.43
             aida04 - 18.82
             aida05 - 20.05
             aida06 - 41.04
             aida07 - 21.24
             aida08 - 205.02 (noisy FEE64)
             aida09 - 24.81
             aida10 - 22.85
             aida11 - 13.98
             aida12 - 19.90

Attachment 5 - New Merger statistics

Attachment 6 - Raw data stream
Attachment 1: 230119_18W.png
230119_18W.png
Attachment 2: 230119_stats.png
230119_stats.png
Attachment 3: 230119_temp.png
230119_temp.png
Attachment 4: 231019_18L.png
231019_18L.png
Attachment 5: 231019_mergestats.png
231019_mergestats.png
Attachment 6: 231019_rawdatastream.png
231019_rawdatastream.png
  16   Wed Jan 23 09:25:55 2019 CA, TD, NH, VPMACB and setup pics 23.01.19
Pictures of leaking cooling pipe

MACB switch settings

AIDA mount - needs rotated

Attachment 11 - AIDA@DESPEC MACB settings - 100MHz and WR
Attachment 1: 50652032_353136001936813_367180692620050432_n.jpg
50652032_353136001936813_367180692620050432_n.jpg
Attachment 2: 50964908_1044865512372636_8341768864786284544_n.jpg
50964908_1044865512372636_8341768864786284544_n.jpg
Attachment 3: 50724736_354107018511390_7832758890239361024_n.jpg
50724736_354107018511390_7832758890239361024_n.jpg
Attachment 4: 50668223_286084358731841_9104059387265155072_n.jpg
50668223_286084358731841_9104059387265155072_n.jpg
Attachment 5: 50628587_236303927288815_6354319211518492672_n.jpg
50628587_236303927288815_6354319211518492672_n.jpg
Attachment 6: 50487774_2016311125120229_9062406111528747008_n.jpg
50487774_2016311125120229_9062406111528747008_n.jpg
Attachment 7: 50458306_365278197604681_5981419301792382976_n.jpg
50458306_365278197604681_5981419301792382976_n.jpg
Attachment 8: 51007009_2128806710538773_6488480452108615680_n.jpg
51007009_2128806710538773_6488480452108615680_n.jpg
Attachment 9: 50309962_360783668035351_2721892157555212288_n.jpg
50309962_360783668035351_2721892157555212288_n.jpg
Attachment 10: 51087974_2761887407370164_7395929517593722880_n.jpg
51087974_2761887407370164_7395929517593722880_n.jpg
Attachment 11: despecmacb.pdf
despecmacb.pdf
  15   Wed Jan 23 08:12:01 2019 CA, TD, NH, VPJanuary 22nd 2019
Attachment 1 - 1.8.W waveform spectra with pulser signal

Attachment 2 - 1.8.L pulser peak spectra

widths: aida01 - 16.06
       
        aida04 - 20.68

        aida05 - 14.03

        aida08 - 73.02

        aida09 - 669.29

        aida11 - 405.35

        aida12 - 397.6

Attachment 3 - good event statistics
               fast comparator threshold (LEC) at 0xff
               FEEs 8,9,11,12 running hot

Attachment 4 - FEE64 temperatures - all ok

System wide checks ok, except aida02 fails ADC calibration

Leak found in seal on cooling pipe, switched power off
Attachment 1: 220119_18W.png
220119_18W.png
Attachment 2: 220119_pulserpeaks.png
220119_pulserpeaks.png
Attachment 3: 220119_stats.png
220119_stats.png
Attachment 4: 220119_Temp.png
220119_Temp.png
  14   Mon Dec 10 11:09:53 2018 CA, TDScintillator pictures/system checks for 07.12.18
11.12 System wide checks ok (attachment 1)
      FEE temperatures ok (attachment 2)
      bias/leakage currents ok (attachment 3)

11.13 1.8.W spectra for naida 1,9,10,11,12 (attachment 4)
      1.8.W spectra expanded in Y direction (attachment 5)
      pulser peaks (attachment 6)
      
11.20 pulser peak widths:
      
      Aida10 -> 98.82

      Aida09 -> 109.17

      Aida11 -> 71.72

      Aida12 -> 92.82

      Aida01 -> 24.05

11.23 hit rates (attachment 7)

11.41 pictures of setup with scintillator mounted (attachment 8-14)
Attachment 1: 071218_stats.png
071218_stats.png
Attachment 2: 071218_temp.png
071218_temp.png
Attachment 3: 071218_bias.png
071218_bias.png
Attachment 4: 071218_18W.png
071218_18W.png
Attachment 5: 071218_18W_Yexp.png
071218_18W_Yexp.png
Attachment 6: 071218_peakwidth.png
071218_peakwidth.png
Attachment 7: 071218_rates.png
071218_rates.png
Attachment 8: 47688070_2256762954565054_3334929654510256128_n.jpg
47688070_2256762954565054_3334929654510256128_n.jpg
Attachment 9: 47305799_497118170799030_7486824778062364672_n.jpg
47305799_497118170799030_7486824778062364672_n.jpg
Attachment 10: 47687230_1459939034137752_2211443514993016832_n.jpg
47687230_1459939034137752_2211443514993016832_n.jpg
Attachment 11: 48336006_599957530448236_3569021065952755712_n.jpg
48336006_599957530448236_3569021065952755712_n.jpg
Attachment 12: 48357475_2127970327241855_1581049677237714944_n.jpg
48357475_2127970327241855_1581049677237714944_n.jpg
Attachment 13: 48192564_414044639133054_7802028369171709952_n.jpg
48192564_414044639133054_7802028369171709952_n.jpg
Attachment 14: 47580605_365365260694379_6209069885788520448_n.jpg
47580605_365365260694379_6209069885788520448_n.jpg
  13   Sat Dec 8 15:22:52 2018 TDRaspberry Pi startup
nnrpi1 - FEE64 system consoles

nnrpi2 - USB-controlled ac mains relay, CAEN N1419ET

username: pi
password: *******

To startup TclHttpd server for web access via port 8015 execute
 
 /MIDAS/TclHttpd/Linux-arm/TclHttpd-server

Update - 9 March 2020 command line should be

/MIDAS/TclHttpd/linux-arm/TclHttpd-server
  12   Thu Dec 6 20:45:47 2018 TDAnalysis of R2_0
Analysis of data file R2_0

Attachment 1 - summary output of TDR analyser

Attachment 2 - verbose output of TDR analyser
Attachment 1: R2_0.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 *** ERROR: READ I/O error:       5002
                   blocks:        211
          ADC data format:     954863 (   22755.4 Hz)
        Other data format:     772383 (   18406.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      10007 (     238.5 Hz)
                              WR48-63:       9384 (     223.6 Hz)
                           FEE64 disc:     752992 (   17944.6 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       41.962 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000              419.598
  2                0.000              419.598
  3                0.000              419.598
  4                0.000              419.598
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0          0          0          0          0          0          0          0          0          0          0          0          0
  1     238554     194165          0          0          0          0       2501       2346     189318          0          0          0
  2     238606     201468          0          0          0          0       2502       2346     196620          0          0          0
  3     238824     203983          0          0          0          0       2502       2346     199135          0          0          0
  4     238879     172767          0          0          0          0       2502       2346     167919          0          0          0
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    0.141s ( 1500.444 blocks/s,  93.778 Mb/s)
Attachment 2: R2_0.txt
[td@localhost ~]$ analyser/analyser v R2_0 | more
 *** TDR format 3.3.0 analyser - TD - May 2018
  verbose
 *** FEE64 discriminator: block:     1 ptr:     7 data: 0x83640100 module:  3 information type:  6 information field: 0x00040100 ts: 0x0000000006332584
 *** FEE64 discriminator: block:     1 ptr:     9 data: 0x84640101 module:  4 information type:  6 information field: 0x00040101 ts: 0x0000000006332584
 *** FEE64 discriminator: block:     1 ptr:    11 data: 0x82610101 module:  2 information type:  6 information field: 0x00010101 ts: 0x0000000006332585
 *** FEE64 discriminator: block:     1 ptr:    13 data: 0x82640101 module:  2 information type:  6 information field: 0x00040101 ts: 0x0000000006332585
 *** FEE64 discriminator: block:     1 ptr:    15 data: 0x83640001 module:  3 information type:  6 information field: 0x00040001 ts: 0x0000000006332585
 ***            ADC data: block:     1 ptr:    17 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x0000000006332C5D
 ***            ADC data: block:     1 ptr:    19 data: 0xC11073EA module:  4 fail: 0 range: 0 id:   272 channel: 16 adc: 29674  ts: 0x0000000006332C5D
 ***            ADC data: block:     1 ptr:    21 data: 0xC120739F module:  4 fail: 0 range: 0 id:   288 channel: 32 adc: 29599  ts: 0x0000000006332C5D
 ***            ADC data: block:     1 ptr:    23 data: 0xC130738F module:  4 fail: 0 range: 0 id:   304 channel: 48 adc: 29583  ts: 0x0000000006332C5D
 ***            ADC data: block:     1 ptr:    25 data: 0xC0C07479 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29817  ts: 0x0000000006332C9E
 ***            ADC data: block:     1 ptr:    27 data: 0xC0D07500 module:  3 fail: 0 range: 0 id:   208 channel: 16 adc: 29952  ts: 0x0000000006332C9E
 ***            ADC data: block:     1 ptr:    29 data: 0xC0E073B3 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 29619  ts: 0x0000000006332C9E
 ***            ADC data: block:     1 ptr:    31 data: 0xC0F07571 module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30065  ts: 0x0000000006332C9E
 ***            ADC data: block:     1 ptr:    33 data: 0xC0407475 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29813  ts: 0x0000000006332CB9
 ***            ADC data: block:     1 ptr:    35 data: 0xC0507482 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29826  ts: 0x0000000006332CB9
 ***            ADC data: block:     1 ptr:    37 data: 0xC06074ED module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29933  ts: 0x0000000006332CB9
 ***            ADC data: block:     1 ptr:    39 data: 0xC0707409 module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29705  ts: 0x0000000006332CB9
 ***            ADC data: block:     1 ptr:    41 data: 0xC108750A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29962  ts: 0x0000000006332D25
 ***            ADC data: block:     1 ptr:    43 data: 0xC1387428 module:  4 fail: 0 range: 0 id:   312 channel: 56 adc: 29736  ts: 0x0000000006332D25
 ***            ADC data: block:     1 ptr:    45 data: 0xC0C8748F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29839  ts: 0x0000000006332D66
 ***            ADC data: block:     1 ptr:    47 data: 0xC0F873EB module:  3 fail: 0 range: 0 id:   248 channel: 56 adc: 29675  ts: 0x0000000006332D66
 ***            ADC data: block:     1 ptr:    49 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x0000000006332D81
 ***            ADC data: block:     1 ptr:    51 data: 0xC07874FF module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29951  ts: 0x0000000006332D81
 *** FEE64 discriminator: block:     1 ptr:    53 data: 0x84620001 module:  4 information type:  6 information field: 0x00020001 ts: 0x0000000006431727
 *** FEE64 discriminator: block:     1 ptr:    55 data: 0x81610001 module:  1 information type:  6 information field: 0x00010001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    57 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    59 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    61 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    63 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    65 data: 0x83620001 module:  3 information type:  6 information field: 0x00020001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    67 data: 0x84610101 module:  4 information type:  6 information field: 0x00010101 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    69 data: 0x84630001 module:  4 information type:  6 information field: 0x00030001 ts: 0x0000000006431728
 *** FEE64 discriminator: block:     1 ptr:    71 data: 0x81610100 module:  1 information type:  6 information field: 0x00010100 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    73 data: 0x81640100 module:  1 information type:  6 information field: 0x00040100 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    75 data: 0x82610001 module:  2 information type:  6 information field: 0x00010001 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    77 data: 0x82620001 module:  2 information type:  6 information field: 0x00020001 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    79 data: 0x82630001 module:  2 information type:  6 information field: 0x00030001 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    81 data: 0x82640101 module:  2 information type:  6 information field: 0x00040101 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    83 data: 0x83610001 module:  3 information type:  6 information field: 0x00010001 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    85 data: 0x83630001 module:  3 information type:  6 information field: 0x00030001 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    87 data: 0x83640101 module:  3 information type:  6 information field: 0x00040101 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    89 data: 0x84640101 module:  4 information type:  6 information field: 0x00040101 ts: 0x0000000006431729
 *** FEE64 discriminator: block:     1 ptr:    91 data: 0x82610100 module:  2 information type:  6 information field: 0x00010100 ts: 0x000000000643172A
 ***            ADC data: block:     1 ptr:    93 data: 0xC10074DC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29916  ts: 0x0000000006431D9D
 ***            ADC data: block:     1 ptr:    95 data: 0xC11073EA module:  4 fail: 0 range: 0 id:   272 channel: 16 adc: 29674  ts: 0x0000000006431D9D
 ***            ADC data: block:     1 ptr:    97 data: 0xC12073AC module:  4 fail: 0 range: 0 id:   288 channel: 32 adc: 29612  ts: 0x0000000006431D9D
 ***            ADC data: block:     1 ptr:    99 data: 0xC1307390 module:  4 fail: 0 range: 0 id:   304 channel: 48 adc: 29584  ts: 0x0000000006431D9D
 ***            ADC data: block:     1 ptr:   101 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x0000000006431DDE
 ***            ADC data: block:     1 ptr:   103 data: 0xC0D07510 module:  3 fail: 0 range: 0 id:   208 channel: 16 adc: 29968  ts: 0x0000000006431DDE
 ***            ADC data: block:     1 ptr:   105 data: 0xC0E073BE module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 29630  ts: 0x0000000006431DDE
 ***            ADC data: block:     1 ptr:   107 data: 0xC0F07570 module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30064  ts: 0x0000000006431DDE
/type:  5
...skipping
 ***   WR48-64 timestamp: block:     1 ptr:  1087 data: 0x81500001 module:  1 information type:  5 information field: 0x00000001 ts: 0x0001000007000000
 ***   SYNC100 timestamp: block:     1 ptr:  1089 data: 0x8141CA02 module:  1 information type:  4 information field: 0x0001CA02 ts: 0x00011CA027000000
 ***   WR48-64 timestamp: block:     1 ptr:  1091 data: 0x82500001 module:  2 information type:  5 information field: 0x00000001 ts: 0x00011CA027000000
 ***   SYNC100 timestamp: block:     1 ptr:  1093 data: 0x8241CA02 module:  2 information type:  4 information field: 0x0001CA02 ts: 0x00011CA027000000
 ***   WR48-64 timestamp: block:     1 ptr:  1095 data: 0x83500001 module:  3 information type:  5 information field: 0x00000001 ts: 0x00011CA027000000
 ***   SYNC100 timestamp: block:     1 ptr:  1097 data: 0x8341CA02 module:  3 information type:  4 information field: 0x0001CA02 ts: 0x00011CA027000000
 ***   WR48-64 timestamp: block:     1 ptr:  1099 data: 0x84500001 module:  4 information type:  5 information field: 0x00000001 ts: 0x00011CA027000000
 ***   SYNC100 timestamp: block:     1 ptr:  1101 data: 0x8441CA02 module:  4 information type:  4 information field: 0x0001CA02 ts: 0x00011CA027000000
 *** FEE64 discriminator: block:     1 ptr:  1103 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1105 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1107 data: 0x81640101 module:  1 information type:  6 information field: 0x00040101 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1109 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1111 data: 0x83620001 module:  3 information type:  6 information field: 0x00020001 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1113 data: 0x84610101 module:  4 information type:  6 information field: 0x00010101 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1115 data: 0x84630001 module:  4 information type:  6 information field: 0x00030001 ts: 0x00011CA027026A19
 *** FEE64 discriminator: block:     1 ptr:  1117 data: 0x82610001 module:  2 information type:  6 information field: 0x00010001 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1119 data: 0x82620001 module:  2 information type:  6 information field: 0x00020001 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1121 data: 0x82630001 module:  2 information type:  6 information field: 0x00030001 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1123 data: 0x82640100 module:  2 information type:  6 information field: 0x00040100 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1125 data: 0x83610001 module:  3 information type:  6 information field: 0x00010001 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1127 data: 0x83630001 module:  3 information type:  6 information field: 0x00030001 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1129 data: 0x83640100 module:  3 information type:  6 information field: 0x00040100 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1131 data: 0x84640101 module:  4 information type:  6 information field: 0x00040101 ts: 0x00011CA027026A1A
 *** FEE64 discriminator: block:     1 ptr:  1133 data: 0x82610100 module:  2 information type:  6 information field: 0x00010100 ts: 0x00011CA027026A1B
 *** FEE64 discriminator: block:     1 ptr:  1135 data: 0x82640001 module:  2 information type:  6 information field: 0x00040001 ts: 0x00011CA027026A1B
 *** FEE64 discriminator: block:     1 ptr:  1137 data: 0x83640001 module:  3 information type:  6 information field: 0x00040001 ts: 0x00011CA027026A1B
 *** FEE64 discriminator: block:     1 ptr:  1139 data: 0x84610100 module:  4 information type:  6 information field: 0x00010100 ts: 0x00011CA027026E68
 ***            ADC data: block:     1 ptr:  1141 data: 0xC06074FD module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29949  ts: 0x00011CA027027049
 ***            ADC data: block:     1 ptr:  1143 data: 0xC080746B module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29803  ts: 0x00011CA027027064
 ***            ADC data: block:     1 ptr:  1145 data: 0xC090755B module:  2 fail: 0 range: 0 id:   144 channel: 16 adc: 30043  ts: 0x00011CA027027064
 ***            ADC data: block:     1 ptr:  1147 data: 0xC0A07458 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29784  ts: 0x00011CA027027064
 ***            ADC data: block:     1 ptr:  1149 data: 0xC0B07507 module:  2 fail: 0 range: 0 id:   176 channel: 48 adc: 29959  ts: 0x00011CA027027064
 ***            ADC data: block:     1 ptr:  1151 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x00011CA0270270C6
 ***            ADC data: block:     1 ptr:  1153 data: 0xC0D07514 module:  3 fail: 0 range: 0 id:   208 channel: 16 adc: 29972  ts: 0x00011CA0270270C6
 ***            ADC data: block:     1 ptr:  1155 data: 0xC0E073BC module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 29628  ts: 0x00011CA0270270C6
 ***            ADC data: block:     1 ptr:  1157 data: 0xC0F0757F module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30079  ts: 0x00011CA0270270C6
 ***            ADC data: block:     1 ptr:  1159 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x00011CA02702712C
 ***            ADC data: block:     1 ptr:  1161 data: 0xC0B87447 module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29767  ts: 0x00011CA02702712C
 ***            ADC data: block:     1 ptr:  1163 data: 0xC10074EE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29934  ts: 0x00011CA02702714D
 ***            ADC data: block:     1 ptr:  1165 data: 0xC11073EC module:  4 fail: 0 range: 0 id:   272 channel: 16 adc: 29676  ts: 0x00011CA02702714D
 ***            ADC data: block:     1 ptr:  1167 data: 0xC12073B1 module:  4 fail: 0 range: 0 id:   288 channel: 32 adc: 29617  ts: 0x00011CA02702714D
 ***            ADC data: block:     1 ptr:  1169 data: 0xC130739A module:  4 fail: 0 range: 0 id:   304 channel: 48 adc: 29594  ts: 0x00011CA02702714D
 ***            ADC data: block:     1 ptr:  1171 data: 0xC0C87499 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29849  ts: 0x00011CA02702718E
 ***            ADC data: block:     1 ptr:  1173 data: 0xC0F873F0 module:  3 fail: 0 range: 0 id:   248 channel: 56 adc: 29680  ts: 0x00011CA02702718E
 ***            ADC data: block:     1 ptr:  1175 data: 0xC0407475 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29813  ts: 0x00011CA0270271D9
 ***            ADC data: block:     1 ptr:  1177 data: 0xC050748B module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29835  ts: 0x00011CA0270271D9
 ***            ADC data: block:     1 ptr:  1179 data: 0xC0707416 module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29718  ts: 0x00011CA0270271D9
 ***            ADC data: block:     1 ptr:  1181 data: 0xC1087511 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29969  ts: 0x00011CA027027215
 ***            ADC data: block:     1 ptr:  1183 data: 0xC1387435 module:  4 fail: 0 range: 0 id:   312 channel: 56 adc: 29749  ts: 0x00011CA027027215
 ***            ADC data: block:     1 ptr:  1185 data: 0xC0487425 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29733  ts: 0x00011CA0270272A1
 ***            ADC data: block:     1 ptr:  1187 data: 0xC078750F module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29967  ts: 0x00011CA0270272A1
 *** FEE64 discriminator: block:     1 ptr:  1189 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x00011CA027125BA2
 *** FEE64 discriminator: block:     1 ptr:  1191 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x00011CA027125BA2

  11   Thu Dec 6 15:31:40 2018 TD, CA, NHDecember 6th 2018
15.32 1.8.W spectra for aida01, aida09, aida10, aida11, aida12 (attachments 1, 2, 3)
      2 and 3 expanded in Y and X direction respectively

15.35 good event statistics (attachment 4)
      FEE temperatures (attachment 5)
      leakage currents/bias (attachment 6)

17.20 attachment 7 - loose ground wire
      attachment 8,9,10 - pictures of wire connections
      attachment 11 - side on view of dsssd mounted in aida snout

17.28 Pulser widths:
      aida10 -> 101.24
      aida09 -> 136.22
      aida11 -> 66.31
      aida12 -> 87.17
      aida01 -> 15.29   
Attachment 1: 061218_18W.png
061218_18W.png
Attachment 2: 061218_18W_expanded.png
061218_18W_expanded.png
Attachment 3: 061218_18W_Xexpand.png
061218_18W_Xexpand.png
Attachment 4: 061218_stats.png
061218_stats.png
Attachment 5: 061218_temp.png
061218_temp.png
Attachment 6: 061218_bias.png
061218_bias.png
Attachment 7: 47093748_2340006526227505_826129468962635776_n.jpg
47093748_2340006526227505_826129468962635776_n.jpg
Attachment 8: 47422277_198613721072538_1679973206540156928_n.jpg
47422277_198613721072538_1679973206540156928_n.jpg
Attachment 9: 47681810_2178074892519875_6296253637383421952_n.jpg
47681810_2178074892519875_6296253637383421952_n.jpg
Attachment 10: 47576820_766724350361835_8170963534457339904_n.jpg
47576820_766724350361835_8170963534457339904_n.jpg
Attachment 11: 47680357_471918063335752_1178331630641610752_n.jpg
47680357_471918063335752_1178331630641610752_n.jpg
  10   Sat Dec 1 16:43:47 2018 TDAnalysis of WRTest28Nov18.dat
Attachment 1 - analyser.f 

Modified version of TDR format analyser program to handle 64 bit WR timestamps

Attachment 2 - analyser program output

12092 blocks = 773888kb

8 timewarps

0 information code 5 data items

data file does not commence with either information codes 5 or 4

times/rates not correctly calculated - program assumes 10ns/tick 

Attachment 3 - od -x output 
Attachment 1: analyser.f
      PROGRAM GREAT

C     Read and analyse GREAT format data - version 3.3.0
C
C     See:
C     http://npg.dl.ac.uk/documents/edoc504/edoc504.html
C     http://npg.dl.ac.uk/DataAcq/TSformat.html

      IMPLICIT NONE                                 
C
C     External functions
C
      EXTERNAL timestamp
C
      DOUBLE PRECISION dfloat
C
      INTEGER and
      INTEGER*8 iargc
      INTEGER rshift
      INTEGER*8 timestamp
C
      REAL secnds
C
C     Parameter variables
C
      INTEGER block_length
      PARAMETER (block_length = 16384)
C
C     Local variables
C
      CHARACTER*255 arg
      CHARACTER*(block_length*4) buffer
      CHARACTER*255 filename
C
      INTEGER adc_data, block_number, block_ptr, ch, channel
      INTEGER channel_ident, data_block(block_length)
      INTEGER*8 dead(32)
      INTEGER event, fail
      INTEGER*8 first_ts
      INTEGER i
      INTEGER*2 i2(2), i2_2(4)
      INTEGER i4, i4_2(2)
      INTEGER*8 i8, idle(32)
      INTEGER information, information_index, io_error, items(0:32,0:31)
      INTEGER j, j4
      INTEGER*8 last_ts
      INTEGER mbs_data, module
      INTEGER*8 pause(32)
      INTEGER range
      INTEGER*8 resume(32)
      INTEGER sample_length, sum(0:31)
      INTEGER*8 sync(32)
      INTEGER tag, tag_old
      INTEGER*8 ts
      INTEGER ts28, ts48, ts64
      INTEGER*8 ts_old
      INTEGER type, type_old
C
      LOGICAL verbose
C
      REAL dt, t1, t2

      EQUIVALENCE ( data_block, buffer )
      EQUIVALENCE ( i4, i2( 1 ) )
      EQUIVALENCE ( i8, i4_2(1) )

      SAVE data_block, block_ptr, block_number, event

C----67---------------------------------------------------------------72------80
C     Announce program

      WRITE( 6, 9000 )

C     Initialise variables

      block_ptr = 1
      block_number = 0
      event = 0

      i4 = 0
      i8 = 0

      ts_old = 0
      tag_old = 0
      type_old = 0
      first_ts = 0
      last_ts = 0
      ts28 = 0
      ts48 = 0
      ts64 = 0

      type = 0
      
      verbose = .FALSE.
      
      DO i = 0, 32
       DO j = 0, 31
        sum( j ) = 0
        items( i, j ) = 0
       ENDDO
      ENDDO

      DO i = 1, 32
       pause( i ) = -1
       resume( i ) = -1
       dead( i ) = 0
       sync( i ) = 0
       idle( i ) = 0
      ENDDO

C     Get input filename from command line argument

      DO i = 1, iargc()
       CALL getarg( i, arg )
       IF ( arg.EQ.'v') THEN
        verbose = .TRUE.
        write(6,*) ' verbose'
       ELSE
        filename = arg
       ENDIF
      ENDDO
      IF ( iargc().LT.1 .OR. iargc().GT.2 ) THEN
       WRITE( 6, * ) ' Usage: ./a.exe [v] <filename>'
       STOP
      ENDIF

      t1 = SECNDS( 0.0 )

C     Open file

      OPEN( 10, RECL = 4 * block_length,
     +      FILE = filename, STATUS = 'OLD', ACCESS = 'DIRECT' )

    1 CONTINUE

C     Read next block
C     Note: for FORTRAN direct I/O first block/record number is 1
C           for GREAT data format first (encoded) block/record number is 0

      block_number = block_number + 1

      READ( 10, REC = block_number, iostat=io_error, ERR=1000 ) buffer

      DO block_ptr = 7, data_block( 6 ) / 4 + 4, 2


       i4 = data_block( block_ptr )
       j4 = data_block( block_ptr + 1 )

       tag = AND( RSHIFT( i4, 30 ), Z'00000003' )

       module = 0
       type = 0

C      tag = 3 = ADC data format
C      tag = 2 = Other data
C      tag = 1 = Sample trace buffer format
C      tag = 0 = undefined?

C----67---------------------------------------------------------------72------80
C     ADC data

       IF ( tag.EQ.3 ) THEN

        fail = AND( RSHIFT( i4, 29 ), Z'00000001' )

        IF ( fail.NE.0 ) THEN

         WRITE( 6, 9600 ) block_number, block_ptr, i4

        ENDIF

        range = AND( RSHIFT( i4, 28 ), Z'00000001' )
        channel_ident = AND( RSHIFT( i4, 16 ), Z'00000fff' )
        module = AND( RSHIFT( channel_ident, 6 ), Z'0000003f' )
        IF ( range.EQ.1 ) THEN
         items(module, 19 ) = items(module, 19 ) + 1
        ENDIF
        channel = AND( channel_ident, Z'0000003f' )
        adc_data = AND( i4, Z'0000ffff' )

        items(module, 0 ) = items(module, 0 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        ts = timestamp( ts64, ts48, ts28, type )

	IF ( ts.LT.ts_old ) THEN

	 items(module, 20 ) = items(module, 20 ) + 1

         WRITE( 6, 9710 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts, ts_old, ts_old-ts,
     +   tag_old, type_old
        ENDIF

        ts_old = ts
        tag_old = tag
        type_old = 0

        IF ( verbose ) THEN 
         WRITE( 6, 9700 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts
        ENDIF

        ch = channel + ( module - 1 ) * 64 + ( range * 2048 )

C----67---------------------------------------------------------------72------80
C     Other information

       ELSEIF( tag.EQ.2 ) THEN

        module = AND( RSHIFT( i4, 24 ), Z'0000003f' )
        type = AND( RSHIFT( i4, 20 ), Z'0000000f' )
        information = AND( i4, Z'000fffff' )

        items(module, 1 ) = items(module, 1 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        IF ( type.EQ.2 ) THEN

C----67---------------------------------------------------------------72------80
C     PAUSE timestamp

         items(module, 4 ) = items(module, 4 ) + 1

        ts48 = information
C        ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 21 ) = items(module, 21 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         pause( module ) = ts

         IF ( verbose ) THEN 
          WRITE( 6, 9800 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.3 ) THEN

C----67---------------------------------------------------------------72------80
C     RESUME timestamp

         items(module, 5 ) = items(module, 5 ) + 1

         ts48 = information
C         ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 22 ) = items(module, 22 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         resume( module ) = ts

         IF ( resume( module ).GE.0 
     +         .AND.
     +        pause( module ).GE.0 
     +          .AND.
     +        resume( module ).GE.pause( module ) ) THEN
          dead( module ) = ( resume( module ) - pause( module ) ) 
     +                     + dead( module )
          pause( module ) = -1
          resume( module ) = -1
         ENDIF

         IF ( verbose ) THEN 
          WRITE( 6, 9810 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.4 ) THEN

C----67---------------------------------------------------------------72------80
C     SYNC100 timestamp/White Rabbit Timestamp Marker (bits 28-47)

         items(module, 6 ) = items(module, 6 ) + 1

         ts48 = information

C        ts = timestamp( ts48, ts28, type )
         ts = timestamp( ts64, ts48, ts28, type )


         IF ( first_ts.EQ.0 ) THEN
          first_ts = ts 
... 446 more lines ...
Attachment 2: WRTest28Nov18.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***   SYNC100 timestamp: block:  3972 ptr:   129 data: 0x824AAF3C module:  2 information type:  4 information field: 0x000AAF3C ts: 0x0000AAF3C0000007 OLD: ts: 0x0000AAF43FB60119 dt: 0x000000007FB60112 tag: 3 type: 0
 ***   SYNC100 timestamp: block:  4117 ptr:    17 data: 0x834AB4EE module:  3 information type:  4 information field: 0x000AB4EE ts: 0x0000AB4EEFFFFFFE OLD: ts: 0x0000AB4F6FE2FB16 dt: 0x000000007FE2FB18 tag: 3 type: 0
 *** FEE64 discriminator: block:  6321 ptr:     7 data: 0x82640001 module:  2 information type:  6 information field: 0x00040001 ts: 0x0000B0BA95E8A4BB OLD: ts: 0x0000B0BA9C641977 dt: 0x00000000067B74BC tag: 3 type: 0
 ***   SYNC100 timestamp: block:  6321 ptr:   359 data: 0x824B0BA3 module:  2 information type:  4 information field: 0x000B0BA3 ts: 0x0000B0BA30000007 OLD: ts: 0x0000B0BA9FD5AFB9 dt: 0x000000006FD5AFB2 tag: 3 type: 0
 *** FEE64 discriminator: block:  8168 ptr:     7 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x0000B545700D8F24 OLD: ts: 0x0000B545737361F0 dt: 0x000000000365D2CC tag: 3 type: 0
 ***   SYNC100 timestamp: block:  8168 ptr:   389 data: 0x834B5451 module:  3 information type:  4 information field: 0x000B5451 ts: 0x0000B54510000006 OLD: ts: 0x0000B5457F907F06 dt: 0x000000006F907F00 tag: 3 type: 0
 ***            ADC data: block:  9068 ptr:     7 data: 0xC0A0746E module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29806  ts: 0x0000B77C0CB89539 OLD: ts: 0x0000B77C0FFFFFFF dt: 0x0000000003476AC6 tag: 2 type: 4
 ***   SYNC100 timestamp: block:  9068 ptr:   125 data: 0x824B77BA module:  2 information type:  4 information field: 0x000B77BA ts: 0x0000B77BA0000001 OLD: ts: 0x0000B77C0FD296D9 dt: 0x000000006FD296D8 tag: 3 type: 0
 *** ERROR: READ I/O error:       5002
                   blocks:      12092
          ADC data format:   54772761 (    1675.2 Hz)
        Other data format:   44212351 (    1352.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:     487184 (      14.9 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:   43725167 (    1337.3 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          1 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          5 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          2 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:    32695.408 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000           326951.701
  2                0.000           327002.704
  3                0.000           326986.598
  4                0.000           326951.701
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0          0          0          0          0          0          0          0          0          0          0          0          0
  1   14927838   11937352          0          0          0          0     121789          0   11815563          0          0          0
  2   14930026   12391697          0          0          0          0     121806          0   12269891          0          0          0
  3    9966853    9183853          0          0          0          0     121802          0    9062051          0          0          0
  4   14948044   10699449          0          0          0          0     121787          0   10577662          0          0          0
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          1          0          0          3          0          1          0          0          0
  3          0          0          0          2          0          1          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    6.453s ( 1873.821 blocks/s, 117.114 Mb/s)
Attachment 3: Capture.PNG
Capture.PNG
  9   Sat Oct 13 17:26:53 2018 TDAnalysis of 3 October 2018 T4 tests
Attachment 1: analyser
Attachment 2: analyser.f
      PROGRAM GREAT

C     Read and analyse GREAT format data - version 3.3.0
C
C     See:
C     http://npg.dl.ac.uk/documents/edoc504/edoc504.html
C     http://npg.dl.ac.uk/DataAcq/TSformat.html

      IMPLICIT NONE                                 
C
C     External functions
C
      EXTERNAL timestamp
C
      DOUBLE PRECISION dfloat
C
      INTEGER and
      INTEGER*8 iargc
      INTEGER rshift
      INTEGER*8 timestamp
C
      REAL secnds
C
C     Parameter variables
C
      INTEGER block_length
      PARAMETER (block_length = 16384)
C
C     Local variables
C
      CHARACTER*255 arg
      CHARACTER*(block_length*4) buffer
      CHARACTER*255 filename
C
      INTEGER adc_data, block_number, block_ptr, ch, channel
      INTEGER channel_ident, data_block(block_length)
      INTEGER*8 dead(32)
      INTEGER event, fail
      INTEGER*8 first_ts
      INTEGER i
      INTEGER*2 i2(2), i2_2(4)
      INTEGER i4, i4_2(2)
      INTEGER*8 i8, idle(32)
      INTEGER information, information_index, io_error, items(0:32,0:31)
      INTEGER j, j4
      INTEGER*8 last_ts
      INTEGER mbs_data, module
      INTEGER*8 pause(32)
      INTEGER range
      INTEGER*8 resume(32)
      INTEGER sample_length, sum(0:31)
      INTEGER*8 sync(32)
      INTEGER tag, tag_old
      INTEGER*8 ts
      INTEGER ts28, ts48, ts64
      INTEGER*8 ts_old
      INTEGER type, type_old
C
      LOGICAL verbose
C
      REAL dt, t1, t2

      EQUIVALENCE ( data_block, buffer )
      EQUIVALENCE ( i4, i2( 1 ) )
      EQUIVALENCE ( i8, i4_2(1) )

      SAVE data_block, block_ptr, block_number, event

C----67---------------------------------------------------------------72------80
C     Announce program

      WRITE( 6, 9000 )

C     Initialise variables

      block_ptr = 1
      block_number = 0
      event = 0

      i4 = 0
      i8 = 0

      ts_old = 0
      tag_old = 0
      type_old = 0
      first_ts = 0
      last_ts = 0
      ts28 = 0
      ts48 = 0
      ts64 = 0

      type = 0
      
      verbose = .FALSE.
      
      DO i = 0, 32
       DO j = 0, 31
        sum( j ) = 0
        items( i, j ) = 0
       ENDDO
      ENDDO

      DO i = 1, 32
       pause( i ) = -1
       resume( i ) = -1
       dead( i ) = 0
       sync( i ) = 0
       idle( i ) = 0
      ENDDO

C     Get input filename from command line argument

      DO i = 1, iargc()
       CALL getarg( i, arg )
       IF ( arg.EQ.'v') THEN
        verbose = .TRUE.
        write(6,*) ' verbose'
       ELSE
        filename = arg
       ENDIF
      ENDDO
      IF ( iargc().LT.1 .OR. iargc().GT.2 ) THEN
       WRITE( 6, * ) ' Usage: ./a.exe [v] <filename>'
       STOP
      ENDIF

      t1 = SECNDS( 0.0 )

C     Open file

      OPEN( 10, RECL = 4 * block_length,
     +      FILE = filename, STATUS = 'OLD', ACCESS = 'DIRECT' )

    1 CONTINUE

C     Read next block
C     Note: for FORTRAN direct I/O first block/record number is 1
C           for GREAT data format first (encoded) block/record number is 0

      block_number = block_number + 1

      READ( 10, REC = block_number, iostat=io_error, ERR=1000 ) buffer

      DO block_ptr = 7, data_block( 6 ) / 4 + 4, 2


       i4 = data_block( block_ptr )
       j4 = data_block( block_ptr + 1 )

       tag = AND( RSHIFT( i4, 30 ), Z'00000003' )

       module = 0
       type = 0

C      tag = 3 = ADC data format
C      tag = 2 = Other data
C      tag = 1 = Sample trace buffer format
C      tag = 0 = undefined?

C----67---------------------------------------------------------------72------80
C     ADC data

       IF ( tag.EQ.3 ) THEN

        fail = AND( RSHIFT( i4, 29 ), Z'00000001' )

        IF ( fail.NE.0 ) THEN

         WRITE( 6, 9600 ) block_number, block_ptr, i4

        ENDIF

        range = AND( RSHIFT( i4, 28 ), Z'00000001' )
        channel_ident = AND( RSHIFT( i4, 16 ), Z'00000fff' )
        module = AND( RSHIFT( channel_ident, 6 ), Z'0000003f' )
        IF ( range.EQ.1 ) THEN
         items(module, 19 ) = items(module, 19 ) + 1
        ENDIF
        channel = AND( channel_ident, Z'0000003f' )
        adc_data = AND( i4, Z'0000ffff' )

        items(module, 0 ) = items(module, 0 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        ts = timestamp( ts64, ts48, ts28, type )

	IF ( ts.LT.ts_old ) THEN

	 items(module, 20 ) = items(module, 20 ) + 1

         WRITE( 6, 9710 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts, ts_old, ts_old-ts,
     +   tag_old, type_old
        ENDIF

        ts_old = ts
        tag_old = tag
        type_old = 0

        IF ( verbose ) THEN 
         WRITE( 6, 9700 ) block_number, block_ptr, i4, module, fail,
     +   range, channel_ident, channel, adc_data, ts
        ENDIF

        ch = channel + ( module - 1 ) * 64 + ( range * 2048 )

C----67---------------------------------------------------------------72------80
C     Other information

       ELSEIF( tag.EQ.2 ) THEN

        module = AND( RSHIFT( i4, 24 ), Z'0000003f' )
        type = AND( RSHIFT( i4, 20 ), Z'0000000f' )
        information = AND( i4, Z'000fffff' )

        items(module, 1 ) = items(module, 1 ) + 1

        ts28 = AND( data_block( block_ptr + 1 ), Z'0fffffff' )

        IF ( type.EQ.2 ) THEN

C----67---------------------------------------------------------------72------80
C     PAUSE timestamp

         items(module, 4 ) = items(module, 4 ) + 1

        ts48 = information
C        ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 21 ) = items(module, 21 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         pause( module ) = ts

         IF ( verbose ) THEN 
          WRITE( 6, 9800 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.3 ) THEN

C----67---------------------------------------------------------------72------80
C     RESUME timestamp

         items(module, 5 ) = items(module, 5 ) + 1

         ts48 = information
C         ts = timestamp( ts48, ts28, type )

         ts = timestamp( ts64, ts48, ts28, type )

         IF ( ts.LT.ts_old ) THEN

          items(module, 22 ) = items(module, 22 ) + 1

         ENDIF
         ts_old = ts
         type_old = type
         tag_old = tag
         resume( module ) = ts

         IF ( resume( module ).GE.0 
     +         .AND.
     +        pause( module ).GE.0 
     +          .AND.
     +        resume( module ).GE.pause( module ) ) THEN
          dead( module ) = ( resume( module ) - pause( module ) ) 
     +                     + dead( module )
          pause( module ) = -1
          resume( module ) = -1
         ENDIF

         IF ( verbose ) THEN 
          WRITE( 6, 9810 ) block_number, block_ptr, i4, module, type,
     +     information, ts
         ENDIF

        ELSEIF( type.EQ.4 ) THEN

C----67---------------------------------------------------------------72------80
C     SYNC100 timestamp/White Rabbit Timestamp Marker (bits 28-47)

         items(module, 6 ) = items(module, 6 ) + 1

         ts48 = information

C        ts = timestamp( ts48, ts28, type )
         ts = timestamp( ts64, ts48, ts28, type )


         IF ( first_ts.EQ.0 ) THEN
          first_ts = ts 
... 446 more lines ...
Attachment 3: FEE0_verbose.out
 ***   WR48-64 timestamp: block:    43 ptr:  1619 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B163AE0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1621 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B163AE0DF
 ***   WR48-64 timestamp: block:    43 ptr:  1623 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B164A80DF
 ***   SYNC100 timestamp: block:    43 ptr:  1625 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B164A80DF
 *** FEE64 discriminator: block:    43 ptr:  1627 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x155A147B16523681
 *** FEE64 discriminator: block:    43 ptr:  1629 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B16523681
 *** FEE64 discriminator: block:    43 ptr:  1631 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x155A147B1652368B
 *** FEE64 discriminator: block:    43 ptr:  1633 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x155A147B1652368B
 *** FEE64 discriminator: block:    43 ptr:  1635 data: 0x81640100 module:  1 information type:  6 information field: 0x00040100 ts: 0x155A147B1652368B
 ***            ADC data: block:    43 ptr:  1637 data: 0xC06074F9 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29945  ts: 0x155A147B1652364F OLD: 
ts: 0x155A147B1652368B dt: 0x000000000000003C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1637 data: 0xC06074F9 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29945  ts: 0x155A147B1652364F
 ***            ADC data: block:    43 ptr:  1639 data: 0xC040747C module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29820  ts: 0x155A147B165245EF
 ***            ADC data: block:    43 ptr:  1641 data: 0xC050748A module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29834  ts: 0x155A147B165245EF
 ***            ADC data: block:    43 ptr:  1643 data: 0xC0707410 module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29712  ts: 0x155A147B165245EF
 ***            ADC data: block:    43 ptr:  1645 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B16522E7F OLD: 
ts: 0x155A147B165245EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1645 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B16522E7F
 ***            ADC data: block:    43 ptr:  1647 data: 0xC07874FE module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29950  ts: 0x155A147B16522E7F
 ***   WR48-64 timestamp: block:    43 ptr:  1649 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B165A20DF
 ***   SYNC100 timestamp: block:    43 ptr:  1651 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B165A20DF
 ***   WR48-64 timestamp: block:    43 ptr:  1653 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B1669C0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1655 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B1669C0DF




 ***   WR48-64 timestamp: block:    43 ptr:  1679 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B167960DF
 ***   SYNC100 timestamp: block:    43 ptr:  1681 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B167960DF
 ***   WR48-64 timestamp: block:    43 ptr:  1683 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B168900DF
 ***   SYNC100 timestamp: block:    43 ptr:  1685 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B168900DF
 *** FEE64 discriminator: block:    43 ptr:  1687 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x155A147B169205BD
 *** FEE64 discriminator: block:    43 ptr:  1689 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x155A147B169205BD
 *** FEE64 discriminator: block:    43 ptr:  1691 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x155A147B169205BD
 *** FEE64 discriminator: block:    43 ptr:  1693 data: 0x81640101 module:  1 information type:  6 information field: 0x00040101 ts: 0x155A147B169205BD
 ***            ADC data: block:    43 ptr:  1695 data: 0xC0407475 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29813  ts: 0x155A147B16922D4F
 ***            ADC data: block:    43 ptr:  1697 data: 0xC0507485 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29829  ts: 0x155A147B16922D4F
 ***            ADC data: block:    43 ptr:  1699 data: 0xC06074F8 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29944  ts: 0x155A147B16922D4F
 ***            ADC data: block:    43 ptr:  1701 data: 0xC070740C module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29708  ts: 0x155A147B16922D4F
 ***            ADC data: block:    43 ptr:  1703 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B169215DF OLD: 
ts: 0x155A147B16922D4F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1703 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B169215DF
 ***            ADC data: block:    43 ptr:  1705 data: 0xC0787507 module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29959  ts: 0x155A147B169215DF
 ***   WR48-64 timestamp: block:    43 ptr:  1707 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B1698A0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1709 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B1698A0DF
 ***   WR48-64 timestamp: block:    43 ptr:  1711 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B16A840DF
 ***   SYNC100 timestamp: block:    43 ptr:  1713 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B16A840DF




 ***   WR48-64 timestamp: block:    43 ptr:  1707 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B1698A0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1709 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B1698A0DF
 ***   WR48-64 timestamp: block:    43 ptr:  1711 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B16A840DF
 ***   SYNC100 timestamp: block:    43 ptr:  1713 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B16A840DF
 *** FEE64 discriminator: block:    43 ptr:  1715 data: 0x81610001 module:  1 information type:  6 information field: 0x00010001 ts: 0x155A147B16B1FA21
 *** FEE64 discriminator: block:    43 ptr:  1717 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x155A147B16B1FA21
 *** FEE64 discriminator: block:    43 ptr:  1719 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x155A147B16B1FA21
 *** FEE64 discriminator: block:    43 ptr:  1721 data: 0x81640101 module:  1 information type:  6 information field: 0x00040101 ts: 0x155A147B16B1FA21
 *** FEE64 discriminator: block:    43 ptr:  1723 data: 0x81610100 module:  1 information type:  6 information field: 0x00010100 ts: 0x155A147B16B1FA2B
 ***            ADC data: block:    43 ptr:  1725 data: 0xC050747C module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29820  ts: 0x155A147B16B2192F
 ***            ADC data: block:    43 ptr:  1727 data: 0xC06074FF module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29951  ts: 0x155A147B16B2192F
 ***            ADC data: block:    43 ptr:  1729 data: 0xC0407477 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29815  ts: 0x155A147B16B2098F OLD: 
ts: 0x155A147B16B2192F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1729 data: 0xC0407477 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29815  ts: 0x155A147B16B2098F
 ***            ADC data: block:    43 ptr:  1731 data: 0xC0707413 module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29715  ts: 0x155A147B16B2098F
 ***            ADC data: block:    43 ptr:  1733 data: 0xC048742B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29739  ts: 0x155A147B16B2115F
 ***            ADC data: block:    43 ptr:  1735 data: 0xC078750C module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29964  ts: 0x155A147B16B2115F
 ***   WR48-64 timestamp: block:    43 ptr:  1737 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B16B7E0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1739 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B16B7E0DF
 ***   WR48-64 timestamp: block:    43 ptr:  1741 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B16C780DF
 ***   SYNC100 timestamp: block:    43 ptr:  1743 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B16C780DF



 ***   WR48-64 timestamp: block:    43 ptr:  1885 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B175420DF
 ***   SYNC100 timestamp: block:    43 ptr:  1887 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B175420DF
 ***   WR48-64 timestamp: block:    43 ptr:  1889 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B1763C0DF
 ***   SYNC100 timestamp: block:    43 ptr:  1891 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B1763C0DF
 *** FEE64 discriminator: block:    43 ptr:  1893 data: 0x81620001 module:  1 information type:  6 information field: 0x00020001 ts: 0x155A147B1771B479
 *** FEE64 discriminator: block:    43 ptr:  1895 data: 0x81630001 module:  1 information type:  6 information field: 0x00030001 ts: 0x155A147B1771B479
 *** FEE64 discriminator: block:    43 ptr:  1897 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B1771B479
 *** FEE64 discriminator: block:    43 ptr:  1899 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x155A147B1771B483
 *** FEE64 discriminator: block:    43 ptr:  1901 data: 0x81640100 module:  1 information type:  6 information field: 0x00040100 ts: 0x155A147B1771B483
 ***            ADC data: block:    43 ptr:  1903 data: 0xC0407479 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29817  ts: 0x155A147B1771DEEF
 ***            ADC data: block:    43 ptr:  1905 data: 0xC050748E module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29838  ts: 0x155A147B1771DEEF
 ***            ADC data: block:    43 ptr:  1907 data: 0xC06074FE module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29950  ts: 0x155A147B1771DEEF
 ***            ADC data: block:    43 ptr:  1909 data: 0xC0707416 module:  1 fail: 0 range: 0 id:   112 channel: 48 adc: 29718  ts: 0x155A147B1771DEEF
 ***            ADC data: block:    43 ptr:  1911 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B1771C77F OLD: 
ts: 0x155A147B1771DEEF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1911 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B1771C77F
 ***            ADC data: block:    43 ptr:  1913 data: 0xC078750C module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29964  ts: 0x155A147B1771C77F
 ***   WR48-64 timestamp: block:    43 ptr:  1915 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B177360DF
 ***   SYNC100 timestamp: block:    43 ptr:  1917 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B177360DF
 ***   WR48-64 timestamp: block:    43 ptr:  1919 data: 0x8150155A module:  1 information type:  5 information field: 0x0000155A ts: 0x155A147B178300DF
 ***   SYNC100 timestamp: block:    43 ptr:  1921 data: 0x814147B1 module:  1 information type:  4 information field: 0x000147B1 ts: 0x155A147B178300DF

Attachment 4: FEE0.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***            ADC data: block:    43 ptr:   603 data: 0xC040747A module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29818  ts: 0x155A147B11F3C5FF OLD: ts: 0x155A147B11F3D063 dt: 0x0000000000000A64 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:   883 data: 0xC0487433 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29747  ts: 0x155A147B1313572F OLD: ts: 0x155A147B13136E9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1073 data: 0xC040746B module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29803  ts: 0x155A147B13F3109F OLD: ts: 0x155A147B13F31617 dt: 0x0000000000000578 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1103 data: 0xC0407480 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29824  ts: 0x155A147B14130C1F OLD: ts: 0x155A147B14130F03 dt: 0x00000000000002E4 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1141 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B1432F02F OLD: ts: 0x155A147B1433079F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1199 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B1472D78F OLD: ts: 0x155A147B1472EEFF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1259 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B14B2BEEF OLD: ts: 0x155A147B14B2D65F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1413 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B1552892F OLD: ts: 0x155A147B1552A09F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1471 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B1592708F OLD: ts: 0x155A147B159287FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1497 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B15B2643F OLD: ts: 0x155A147B15B27BAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1637 data: 0xC06074F9 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29945  ts: 0x155A147B1652364F OLD: ts: 0x155A147B1652368B dt: 0x000000000000003C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1645 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B16522E7F OLD: ts: 0x155A147B165245EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1703 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B169215DF OLD: ts: 0x155A147B16922D4F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1729 data: 0xC0407477 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29815  ts: 0x155A147B16B2098F OLD: ts: 0x155A147B16B2192F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1911 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B1771C77F OLD: ts: 0x155A147B1771DEEF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1971 data: 0xC048742F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29743  ts: 0x155A147B17B1AEDF OLD: ts: 0x155A147B17B1C64F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2181 data: 0xC0487425 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29733  ts: 0x155A147B1891607F OLD: ts: 0x155A147B189177EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2387 data: 0xC048741A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29722  ts: 0x155A147B1971121F OLD: ts: 0x155A147B1971298F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2447 data: 0xC0487434 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29748  ts: 0x155A147B19B0F97F OLD: ts: 0x155A147B19B110EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2527 data: 0xC0607500 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29952  ts: 0x155A147B1A10D48F OLD: ts: 0x155A147B1A10D4DF dt: 0x0000000000000050 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2589 data: 0xC040748C module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29836  ts: 0x155A147B1A50CB8F OLD: ts: 0x155A147B1A50CF9F dt: 0x0000000000000410 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2619 data: 0xC06074FF module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29951  ts: 0x155A147B1A70B76F OLD: ts: 0x155A147B1A70B7B5 dt: 0x0000000000000046 tag: 2 type: 6
 *** FEE64 discriminator: block:    43 ptr:  2653 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B1A90A223 OLD: ts: 0x155A147B1A90B3C1 dt: 0x000000000000119E tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2663 data: 0xC0487431 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29745  ts: 0x155A147B1A90AB1F OLD: ts: 0x155A147B1A90C28F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2719 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B1AD0927F OLD: ts: 0x155A147B1AD0A9EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2919 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B1BB0441F OLD: ts: 0x155A147B1BB05B8F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2979 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B1BF02B7F OLD: ts: 0x155A147B1BF042EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3129 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B1C8FF5BF OLD: ts: 0x155A147B1C900D2F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3401 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B1DAF8EBF OLD: ts: 0x155A147B1DAFA62F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3455 data: 0xC0407478 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29816  ts: 0x155A147B1DEF7DEF OLD: ts: 0x155A147B1DEF8411 dt: 0x0000000000000622 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3611 data: 0xC0487433 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29747  ts: 0x155A147B1E8F405F OLD: ts: 0x155A147B1E8F57CF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3819 data: 0xC048742F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29743  ts: 0x155A147B1F6EF1FF OLD: ts: 0x155A147B1F6F096F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3997 data: 0xC040747F module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29823  ts: 0x155A147B202EAFEF OLD: ts: 0x155A147B202EBD23 dt: 0x0000000000000D34 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  4087 data: 0xC048742F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29743  ts: 0x155A147B208E8AFF OLD: ts: 0x155A147B208EA26F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   137 data: 0xC0487438 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29752  ts: 0x155A147B216E3C9F OLD: ts: 0x155A147B216E540F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   309 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B222DFA8F OLD: ts: 0x155A147B222E11FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   371 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B226DE1EF OLD: ts: 0x155A147B226DF95F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   431 data: 0xC0487430 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29744  ts: 0x155A147B22ADC94F OLD: ts: 0x155A147B22ADE0BF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   639 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B238D7AEF OLD: ts: 0x155A147B238D925F dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    44 ptr:   747 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B240D4739 OLD: ts: 0x155A147B240D5B25 dt: 0x00000000000013EC tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   781 data: 0xC040747E module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29822  ts: 0x155A147B242D4CFF OLD: ts: 0x155A147B242D5515 dt: 0x0000000000000816 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   849 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B246D2C8F OLD: ts: 0x155A147B246D43FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   891 data: 0xC0407480 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29824  ts: 0x155A147B24AD1BBF OLD: ts: 0x155A147B24AD2B5F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   921 data: 0xC040748E module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29838  ts: 0x155A147B24CD173F OLD: ts: 0x155A147B24CD1C17 dt: 0x00000000000004D8 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1047 data: 0xC048741F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29727  ts: 0x155A147B254CDE2F OLD: ts: 0x155A147B254CF59F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1251 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B262C8FCF OLD: ts: 0x155A147B262CA73F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1311 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B266C772F OLD: ts: 0x155A147B266C8E9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1511 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B274C28CF OLD: ts: 0x155A147B274C403F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1651 data: 0xC060750D module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29965  ts: 0x155A147B27EBFADF OLD: ts: 0x155A147B27EBFB39 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1659 data: 0xC048742F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29743  ts: 0x155A147B27EBF30F OLD: ts: 0x155A147B27EC0A7F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1717 data: 0xC0487436 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29750  ts: 0x155A147B282BDA6F OLD: ts: 0x155A147B282BF1DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1857 data: 0xC0407480 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29824  ts: 0x155A147B28CBAC7F OLD: ts: 0x155A147B28CBAF77 dt: 0x00000000000002F8 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1921 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B290B8C0F OLD: ts: 0x155A147B290BA37F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1967 data: 0xC0407492 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29842  ts: 0x155A147B294B7B3F OLD: ts: 0x155A147B294B8ADF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2025 data: 0xC0607515 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29973  ts: 0x155A147B298B629F OLD: ts: 0x155A147B298B6321 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2063 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B29AB564F OLD: ts: 0x155A147B29AB6DBF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2181 data: 0xC048741B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29723  ts: 0x155A147B2A2B250F OLD: ts: 0x155A147B2A2B3C7F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2237 data: 0xC0787504 module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29956  ts: 0x155A147B2A6B0C6F OLD: ts: 0x155A147B2A6B23DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2319 data: 0xC06074F8 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29944  ts: 0x155A147B2ACAE77F OLD: ts: 0x155A147B2ACAE7CF dt: 0x0000000000000050 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2437 data: 0xC048743C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29756  ts: 0x155A147B2B4ABE0F OLD: ts: 0x155A147B2B4AD57F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2497 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B2B8AA56F OLD: ts: 0x155A147B2B8ABCDF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2609 data: 0xC040747D module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29821  ts: 0x155A147B2C0A7BFF OLD: ts: 0x155A147B2C0A8B9F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2641 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B2C2A6FAF OLD: ts: 0x155A147B2C2A871F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2727 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B2C8A528F OLD: ts: 0x155A147B2C8A69FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2785 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B2CCA39EF OLD: ts: 0x155A147B2CCA515F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3021 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B2DC9DF3F OLD: ts: 0x155A147B2DC9F6AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3229 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B2EA990DF OLD: ts: 0x155A147B2EA9A84F dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    44 ptr:  3535 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x155A147B200903F7 OLD: ts: 0x155A147B2FFCA0DF dt: 0x000000000FF39CE8 tag: 2 type: 4
 ***            ADC data: block:    44 ptr:  3549 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B2009113F OLD: ts: 0x155A147B200928AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3669 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B3088DFFF OLD: ts: 0x155A147B3088F76F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3695 data: 0xC0607513 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29971  ts: 0x155A147B30A8D3AF OLD: ts: 0x155A147B30A8D48B dt: 0x00000000000000DC tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3819 data: 0xC0487422 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29730  ts: 0x155A147B3128AA3F OLD: ts: 0x155A147B3128C1AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4057 data: 0xC060750B module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29963  ts: 0x155A147B32484B0F OLD: ts: 0x155A147B32484C27 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4065 data: 0xC0487437 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29751  ts: 0x155A147B3248433F OLD: ts: 0x155A147B32485AAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4123 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B32882A9F OLD: ts: 0x155A147B3288420F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4183 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B32C811FF OLD: ts: 0x155A147B32C8296F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4593 data: 0xC048741D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29725  ts: 0x155A147B3487753F OLD: ts: 0x155A147B34878CAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4653 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B34C75C9F OLD: ts: 0x155A147B34C7740F dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    44 ptr:  4793 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B35671C53 OLD: ts: 0x155A147B35672ECD dt: 0x000000000000127A tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4805 data: 0xC0487429 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29737  ts: 0x155A147B356726DF OLD: ts: 0x155A147B35673E4F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4863 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B35A70E3F OLD: ts: 0x155A147B35A725AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4923 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B35E6F59F OLD: ts: 0x155A147B35E70D0F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4979 data: 0xC0407474 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29812  ts: 0x155A147B3626E4CF OLD: ts: 0x155A147B3626E96B dt: 0x000000000000049C tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5077 data: 0xC0487425 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29733  ts: 0x155A147B3686BFDF OLD: ts: 0x155A147B3686D74F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5137 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B36C6A73F OLD: ts: 0x155A147B36C6BEAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5287 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B3766717F OLD: ts: 0x155A147B376688EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5341 data: 0xC07874FD module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29949  ts: 0x155A147B37A658DF OLD: ts: 0x155A147B37A6704F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5451 data: 0xC060750B module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29963  ts: 0x155A147B38262F6F OLD: ts: 0x155A147B38263019 dt: 0x00000000000000AA tag: 2 type: 6
 *** FEE64 discriminator: block:    44 ptr:  5539 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B38860011 OLD: ts: 0x155A147B388611E1 dt: 0x00000000000011D0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5609 data: 0xC0487417 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29719  ts: 0x155A147B38C5F1DF OLD: ts: 0x155A147B38C6094F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:    35 data: 0xC0407472 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29810  ts: 0x155A147B3905E10F OLD: ts: 0x155A147B3905F0AF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   155 data: 0xC048741D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29725  ts: 0x155A147B3985AFCF OLD: ts: 0x155A147B3985C73F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   215 data: 0xC0487437 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29751  ts: 0x155A147B39C5972F OLD: ts: 0x155A147B39C5AE9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   269 data: 0xC040747D module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29821  ts: 0x155A147B3A05865F OLD: ts: 0x155A147B3A0595FF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   321 data: 0xC06074EE module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29934  ts: 0x155A147B3A456DBF OLD: ts: 0x155A147B3A456EA5 dt: 0x00000000000000E6 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   349 data: 0xC060750F module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29967  ts: 0x155A147B3A65693F OLD: ts: 0x155A147B3A6569CB dt: 0x000000000000008C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   357 data: 0xC048743D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29757  ts: 0x155A147B3A65616F OLD: ts: 0x155A147B3A6578DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   435 data: 0xC0607500 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29952  ts: 0x155A147B3AC53C7F OLD: ts: 0x155A147B3AC53CF7 dt: 0x0000000000000078 tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:   523 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B3B25137F OLD: ts: 0x155A147B3B2527ED dt: 0x000000000000146E tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   561 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B3B45130F OLD: ts: 0x155A147B3B452A7F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   643 data: 0xC060750E module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29966  ts: 0x155A147B3BA4EE1F OLD: ts: 0x155A147B3BA4EF2D dt: 0x000000000000010E tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   769 data: 0xC048741D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29725  ts: 0x155A147B3C24C4AF OLD: ts: 0x155A147B3C24DC1F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   971 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B3D04764F OLD: ts: 0x155A147B3D048DBF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1027 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B3D445DAF OLD: ts: 0x155A147B3D44751F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1079 data: 0xC0507492 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29842  ts: 0x155A147B3D844CDF OLD: ts: 0x155A147B3D844D39 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1177 data: 0xC0487431 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29745  ts: 0x155A147B3DE427EF OLD: ts: 0x155A147B3DE43F5F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1387 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B3EC3D98F OLD: ts: 0x155A147B3EC3F0FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1447 data: 0xC0487436 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29750  ts: 0x155A147B3F03C0EF OLD: ts: 0x155A147B3F03D85F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1653 data: 0xC048742B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29739  ts: 0x155A147B3FE3728F OLD: ts: 0x155A147B3FE389FF dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    45 ptr:  1665 data: 0x81610101 module:  1 information type:  6 information field: 0x00010101 ts: 0x155A147B30035B27 OLD: ts: 0x155A147B3FFA60DF dt: 0x000000000FF705B8 tag: 2 type: 4
 ***            ADC data: block:    45 ptr:  1853 data: 0xC0487434 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29748  ts: 0x155A147B40C3242F OLD: ts: 0x155A147B40C33B9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1873 data: 0xC0407480 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29824  ts: 0x155A147B40E317DF OLD: ts: 0x155A147B40E31893 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2059 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B41A2D5CF OLD: ts: 0x155A147B41A2ED3F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2121 data: 0xC0487437 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29751  ts: 0x155A147B41E2BD2F OLD: ts: 0x155A147B41E2D49F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2145 data: 0xC0407491 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29841  ts: 0x155A147B4202B0DF OLD: ts: 0x155A147B4202C07F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2179 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B4222A48F OLD: ts: 0x155A147B4222BBFF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2451 data: 0xC06074FB module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29947  ts: 0x155A147B436240DF OLD: ts: 0x155A147B4362419D dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2459 data: 0xC0487429 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29737  ts: 0x155A147B4362390F OLD: ts: 0x155A147B4362507F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2535 data: 0xC0607509 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29961  ts: 0x155A147B43C2141F OLD: ts: 0x155A147B43C214E7 dt: 0x00000000000000C8 tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  2599 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B4401F7AB OLD: ts: 0x155A147B440209D5 dt: 0x000000000000122A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2727 data: 0xC0487431 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29745  ts: 0x155A147B4481D20F OLD: ts: 0x155A147B4481E97F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2809 data: 0xC0407476 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29814  ts: 0x155A147B44E1BCBF OLD: ts: 0x155A147B44E1C2A5 dt: 0x00000000000005E6 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2875 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B45219C4F OLD: ts: 0x155A147B4521B3BF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3055 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B45E15A3F OLD: ts: 0x155A147B45E171AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3115 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B4621419F OLD: ts: 0x155A147B4621590F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3171 data: 0xC0407490 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29840  ts: 0x155A147B466130CF OLD: ts: 0x155A147B4661406F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3323 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B4700F33F OLD: ts: 0x155A147B47010AAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3379 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B4740DA9F OLD: ts: 0x155A147B4740F20F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3585 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B48208C3F OLD: ts: 0x155A147B4820A3AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3609 data: 0xC040746E module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29806  ts: 0x155A147B48407FEF OLD: ts: 0x155A147B48408F8F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3643 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B4860739F OLD: ts: 0x155A147B48608B0F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3785 data: 0xC048741D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29725  ts: 0x155A147B49003DDF OLD: ts: 0x155A147B4900554F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3899 data: 0xC040748C module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29836  ts: 0x155A147B4980146F OLD: ts: 0x155A147B49801E0B dt: 0x000000000000099C tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  3973 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B49DFE4A3 OLD: ts: 0x155A147B49DFF853 dt: 0x00000000000013B0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3983 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B49DFEF7F OLD: ts: 0x155A147B49E006EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4043 data: 0xC0487416 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29718  ts: 0x155A147B4A1FD6DF OLD: ts: 0x155A147B4A1FEE4F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4251 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B4AFF887F OLD: ts: 0x155A147B4AFF9FEF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4457 data: 0xC0487434 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29748  ts: 0x155A147B4BDF3A1F OLD: ts: 0x155A147B4BDF518F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4515 data: 0xC0487425 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29733  ts: 0x155A147B4C1F217F OLD: ts: 0x155A147B4C1F38EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4573 data: 0xC0407471 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29809  ts: 0x155A147B4C5F10AF OLD: ts: 0x155A147B4C5F204F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4697 data: 0xC048741F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29727  ts: 0x155A147B4CDED79F OLD: ts: 0x155A147B4CDEEF0F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4721 data: 0xC0407496 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29846  ts: 0x155A147B4CFECB4F OLD: ts: 0x155A147B4CFED31F dt: 0x00000000000007D0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4965 data: 0xC0487446 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29766  ts: 0x155A147B4DFE709F OLD: ts: 0x155A147B4DFE880F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5073 data: 0xC060750C module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29964  ts: 0x155A147B4E7E472F OLD: ts: 0x155A147B4E7E47ED dt: 0x00000000000000BE tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  5133 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B4EBE247B OLD: ts: 0x155A147B4EBE360F dt: 0x0000000000001194 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5175 data: 0xC0487435 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29749  ts: 0x155A147B4EDE223F OLD: ts: 0x155A147B4EDE39AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5347 data: 0xC06074F7 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29943  ts: 0x155A147B4F9DE02F OLD: ts: 0x155A147B4F9DE07F dt: 0x0000000000000050 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5401 data: 0xC0607508 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29960  ts: 0x155A147B4FDDC78F OLD: ts: 0x155A147B4FDDC82F dt: 0x00000000000000A0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5499 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B503DA29F OLD: ts: 0x155A147B503DBA0F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5559 data: 0xC0787509 module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29961  ts: 0x155A147B507D89FF OLD: ts: 0x155A147B507DA16F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   139 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B511D543F OLD: ts: 0x155A147B511D6BAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   229 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B517D2F4F OLD: ts: 0x155A147B517D46BF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   345 data: 0xC06074FD module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29949  ts: 0x155A147B51FD05DF OLD: ts: 0x155A147B51FD13C7 dt: 0x0000000000000DE8 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   441 data: 0xC0487434 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29748  ts: 0x155A147B525CE0EF OLD: ts: 0x155A147B525CF85F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   465 data: 0xC0407478 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29816  ts: 0x155A147B527CD49F OLD: ts: 0x155A147B527CE43F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   549 data: 0xC0607506 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29958  ts: 0x155A147B52DCB77F OLD: ts: 0x155A147B52DCB847 dt: 0x00000000000000C8 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   643 data: 0xC048742B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29739  ts: 0x155A147B533C928F OLD: ts: 0x155A147B533CA9FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   867 data: 0xC0407308 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29448  ts: 0x155A147B543C37DF OLD: ts: 0x155A147B543C3F0F dt: 0x0000000000000730 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   901 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B545C2B8F OLD: ts: 0x155A147B545C42FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1169 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B557BC48F OLD: ts: 0x155A147B557BDBFF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1315 data: 0xC048742B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29739  ts: 0x155A147B561B8ECF OLD: ts: 0x155A147B561BA63F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1335 data: 0xC0407479 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29817  ts: 0x155A147B563B827F OLD: ts: 0x155A147B563B921F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1371 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B565B762F OLD: ts: 0x155A147B565B8D9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1513 data: 0xC048741F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29727  ts: 0x155A147B56FB406F OLD: ts: 0x155A147B56FB57DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1573 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B573B27CF OLD: ts: 0x155A147B573B3F3F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1599 data: 0xC0407477 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29815  ts: 0x155A147B575B1B7F OLD: ts: 0x155A147B575B2B1F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1835 data: 0xC0607508 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29960  ts: 0x155A147B585AC89F OLD: ts: 0x155A147B585AC95D dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1843 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B585AC0CF OLD: ts: 0x155A147B585AD83F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1899 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B589AA82F OLD: ts: 0x155A147B589ABF9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1953 data: 0xC0407482 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29826  ts: 0x155A147B58DA975F OLD: ts: 0x155A147B58DAA6FF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2119 data: 0xC06074F7 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29943  ts: 0x155A147B599A4D7F OLD: ts: 0x155A147B599A4DF7 dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2147 data: 0xC0407489 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29833  ts: 0x155A147B59BA48FF OLD: ts: 0x155A147B59BA4E09 dt: 0x000000000000050A tag: 2 type: 6
 *** FEE64 discriminator: block:    46 ptr:  2267 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B5A3A0667 OLD: ts: 0x155A147B5A3A18F5 dt: 0x000000000000128E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2277 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B5A3A0FEF OLD: ts: 0x155A147B5A3A275F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2335 data: 0xC0487417 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29719  ts: 0x155A147B5A79F74F OLD: ts: 0x155A147B5A7A0EBF dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    46 ptr:  2495 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B5B39AB71 OLD: ts: 0x155A147B5B39BD7D dt: 0x000000000000120C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2535 data: 0xC0487431 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29745  ts: 0x155A147B5B59A8EF OLD: ts: 0x155A147B5B59C05F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2591 data: 0xC0487441 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29761  ts: 0x155A147B5B99904F OLD: ts: 0x155A147B5B99A7BF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2671 data: 0xC0607505 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29957  ts: 0x155A147B5BF96B5F OLD: ts: 0x155A147B5BF96C13 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2795 data: 0xC048742B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29739  ts: 0x155A147B5C7941EF OLD: ts: 0x155A147B5C79595F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2817 data: 0xC060750A module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29962  ts: 0x155A147B5C99359F OLD: ts: 0x155A147B5C99365D dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2879 data: 0xC040748F module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29839  ts: 0x155A147B5CD92C9F OLD: ts: 0x155A147B5CD92DB7 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2945 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B5D190C2F OLD: ts: 0x155A147B5D19239F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2969 data: 0xC0407489 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29833  ts: 0x155A147B5D3907AF OLD: ts: 0x155A147B5D39174F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3059 data: 0xC040748E module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29838  ts: 0x155A147B5D98DAEF OLD: ts: 0x155A147B5D98E409 dt: 0x000000000000091A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3145 data: 0xC0507480 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29824  ts: 0x155A147B5DF8BDCF OLD: ts: 0x155A147B5DF8BDED dt: 0x000000000000001E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3237 data: 0xC0487433 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29747  ts: 0x155A147B5E5898DF OLD: ts: 0x155A147B5E58B04F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3259 data: 0xC0407470 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29808  ts: 0x155A147B5E788C8F OLD: ts: 0x155A147B5E788F87 dt: 0x00000000000002F8 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3319 data: 0xC06074ED module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29933  ts: 0x155A147B5EB873EF OLD: ts: 0x155A147B5EB8747B dt: 0x000000000000008C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3505 data: 0xC0487417 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29719  ts: 0x155A147B5F7831DF OLD: ts: 0x155A147B5F78494F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3715 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B6057E37F OLD: ts: 0x155A147B6057FAEF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3859 data: 0xC048741B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29723  ts: 0x155A147B60F7ADBF OLD: ts: 0x155A147B60F7C52F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3881 data: 0xC040748D module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29837  ts: 0x155A147B6117A16F OLD: ts: 0x155A147B6117B10F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3937 data: 0xC0607507 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29959  ts: 0x155A147B615788CF OLD: ts: 0x155A147B6157891F dt: 0x0000000000000050 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4203 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B6277299F OLD: ts: 0x155A147B6277410F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4261 data: 0xC048741A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29722  ts: 0x155A147B62B710FF OLD: ts: 0x155A147B62B7286F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4317 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B62F6F85F OLD: ts: 0x155A147B62F70FCF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4401 data: 0xC0407488 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29832  ts: 0x155A147B6356D36F OLD: ts: 0x155A147B6356E053 dt: 0x0000000000000CE4 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4525 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B63D6A9FF OLD: ts: 0x155A147B63D6C16F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4673 data: 0xC0487437 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29751  ts: 0x155A147B6476743F OLD: ts: 0x155A147B64768BAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4731 data: 0xC048742E module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29742  ts: 0x155A147B64B65B9F OLD: ts: 0x155A147B64B6730F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4751 data: 0xC0407493 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29843  ts: 0x155A147B64D64F4F OLD: ts: 0x155A147B64D65EEF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4787 data: 0xC048741F module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29727  ts: 0x155A147B64F642FF OLD: ts: 0x155A147B64F65A6F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5047 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B6615DBFF OLD: ts: 0x155A147B6615F36F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5245 data: 0xC060750B module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29963  ts: 0x155A147B66F5956F OLD: ts: 0x155A147B66F595B5 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5253 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B66F58D9F OLD: ts: 0x155A147B66F5A50F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5313 data: 0xC0487429 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29737  ts: 0x155A147B673574FF OLD: ts: 0x155A147B67358C6F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5523 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B6815269F OLD: ts: 0x155A147B68153E0F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5635 data: 0xC0407471 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29809  ts: 0x155A147B6894FD2F OLD: ts: 0x155A147B689507C5 dt: 0x0000000000000A96 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:    99 data: 0xC0487430 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29744  ts: 0x155A147B68F4D83F OLD: ts: 0x155A147B68F4EFAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   123 data: 0xC0407481 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29825  ts: 0x155A147B6914CBEF OLD: ts: 0x155A147B6914DB8F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   279 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B69B4962F OLD: ts: 0x155A147B69B4AD9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   393 data: 0xC0407485 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29829  ts: 0x155A147B6A347C5F OLD: ts: 0x155A147B6A347E2B dt: 0x00000000000001CC tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   429 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B6A54606F OLD: ts: 0x155A147B6A5477DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   491 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B6A9447CF OLD: ts: 0x155A147B6A945F3F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   551 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B6AD42F2F OLD: ts: 0x155A147B6AD4469F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   751 data: 0xC0607508 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29960  ts: 0x155A147B6BB3E89F OLD: ts: 0x155A147B6BB3E903 dt: 0x0000000000000064 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   759 data: 0xC048743A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29754  ts: 0x155A147B6BB3E0CF OLD: ts: 0x155A147B6BB3F83F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   819 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B6BF3C82F OLD: ts: 0x155A147B6BF3DF9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   965 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B6C93926F OLD: ts: 0x155A147B6C93A9DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1233 data: 0xC0487437 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29751  ts: 0x155A147B6DB32B6F OLD: ts: 0x155A147B6DB342DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1441 data: 0xC048742D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29741  ts: 0x155A147B6E92DD0F OLD: ts: 0x155A147B6E92F47F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1589 data: 0xC0487419 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29721  ts: 0x155A147B6F32A74F OLD: ts: 0x155A147B6F32BEBF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1647 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B6F728EAF OLD: ts: 0x155A147B6F72A61F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1669 data: 0xC060750E module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29966  ts: 0x155A147B6F92825F OLD: ts: 0x155A147B6F9282E1 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1701 data: 0xC06074FD module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29949  ts: 0x155A147B6FB27DDF OLD: ts: 0x155A147B6FB27E25 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1731 data: 0xC0507495 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29845  ts: 0x155A147B6FD2795F OLD: ts: 0x155A147B6FD27973 dt: 0x0000000000000014 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1741 data: 0xC0487439 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29753  ts: 0x155A147B6FD2718F OLD: ts: 0x155A147B6FD288FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1769 data: 0xC0407479 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29817  ts: 0x155A147B6FF2653F OLD: ts: 0x155A147B6FF27CAF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1799 data: 0xC0487436 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29750  ts: 0x155A147B701258EF OLD: ts: 0x155A147B7012705F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1859 data: 0xC048741D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29725  ts: 0x155A147B7052404F OLD: ts: 0x155A147B705257BF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1881 data: 0xC060750F module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29967  ts: 0x155A147B707233FF OLD: ts: 0x155A147B70723517 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1909 data: 0xC06074FC module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29948  ts: 0x155A147B70922F7F OLD: ts: 0x155A147B70922FC5 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2005 data: 0xC0487434 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29748  ts: 0x155A147B70F20A8F OLD: ts: 0x155A147B70F221FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2061 data: 0xC0487442 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29762  ts: 0x155A147B7131F1EF OLD: ts: 0x155A147B7132095F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2211 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B71D1BC2F OLD: ts: 0x155A147B71D1D39F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2271 data: 0xC048741B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29723  ts: 0x155A147B7211A38F OLD: ts: 0x155A147B7211BAFF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2349 data: 0xC0607510 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29968  ts: 0x155A147B72717E9F OLD: ts: 0x155A147B72717F3F dt: 0x00000000000000A0 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2463 data: 0xC048743D module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29757  ts: 0x155A147B72F1552F OLD: ts: 0x155A147B72F16C9F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2613 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B73911F6F OLD: ts: 0x155A147B739136DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2723 data: 0xC0407472 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29810  ts: 0x155A147B7410F5FF OLD: ts: 0x155A147B7410FCFD dt: 0x00000000000006FE tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2781 data: 0xC0607506 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29958  ts: 0x155A147B7450DD5F OLD: ts: 0x155A147B7450DE09 dt: 0x00000000000000AA tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2811 data: 0xC0507490 module:  1 fail: 0 range: 0 id:    80 channel: 16 adc: 29840  ts: 0x155A147B7470D8DF OLD: ts: 0x155A147B7470D907 dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2821 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B7470D10F OLD: ts: 0x155A147B7470E87F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2875 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B74B0B86F OLD: ts: 0x155A147B74B0CFDF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3321 data: 0xC0487422 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29730  ts: 0x155A147B76900F5F OLD: ts: 0x155A147B769026CF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3381 data: 0xC048741E module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29726  ts: 0x155A147B76CFF6BF OLD: ts: 0x155A147B76D00E2F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3457 data: 0xC06074F0 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29936  ts: 0x155A147B772FD1CF OLD: ts: 0x155A147B772FD251 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  3489 data: 0xC060750C module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29964  ts: 0x155A147B774FCD4F OLD: ts: 0x155A147B774FCE5D dt: 0x000000000000010E tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  3585 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B77AFA85F OLD: ts: 0x155A147B77AFBFCF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3845 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B78CF415F OLD: ts: 0x155A147B78CF58CF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3905 data: 0xC048742A module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29738  ts: 0x155A147B790F28BF OLD: ts: 0x155A147B790F402F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4159 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B7A2EC1BF OLD: ts: 0x155A147B7A2ED92F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4219 data: 0xC0487429 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29737  ts: 0x155A147B7A6EA91F OLD: ts: 0x155A147B7A6EC08F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4299 data: 0xC0407478 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29816  ts: 0x155A147B7ACE93CF OLD: ts: 0x155A147B7ACE961D dt: 0x000000000000024E tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  4365 data: 0xC048742C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29740  ts: 0x155A147B7B0E735F OLD: ts: 0x155A147B7B0E8ACF dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    47 ptr:  4513 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B7BAE36E7 OLD: ts: 0x155A147B7BAE4817 dt: 0x0000000000001130 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  4523 data: 0xC048742E module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29742  ts: 0x155A147B7BAE3D9F OLD: ts: 0x155A147B7BAE550F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4573 data: 0xC040748C module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29836  ts: 0x155A147B7BEE2CCF OLD: ts: 0x155A147B7BEE2F13 dt: 0x0000000000000244 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  4729 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B7C8DEF3F OLD: ts: 0x155A147B7C8E06AF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4931 data: 0xC0487421 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29729  ts: 0x155A147B7D6DA0DF OLD: ts: 0x155A147B7D6DB84F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5141 data: 0xC0487426 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29734  ts: 0x155A147B7E4D527F OLD: ts: 0x155A147B7E4D69EF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5289 data: 0xC0487420 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29728  ts: 0x155A147B7EED1CBF OLD: ts: 0x155A147B7EED342F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5349 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B7F2D041F OLD: ts: 0x155A147B7F2D1B8F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5369 data: 0xC0407494 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29844  ts: 0x155A147B7F4CF7CF OLD: ts: 0x155A147B7F4D0107 dt: 0x0000000000000938 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  5459 data: 0xC0407491 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29841  ts: 0x155A147B7FACDAAF OLD: ts: 0x155A147B7FACE54F dt: 0x0000000000000AA0 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  5605 data: 0xC0487440 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29760  ts: 0x155A147B804C9D1F OLD: ts: 0x155A147B804CB48F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:    27 data: 0xC0407481 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29825  ts: 0x155A147B808C8C4F OLD: ts: 0x155A147B808C8FA1 dt: 0x0000000000000352 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:   155 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B810C5B0F OLD: ts: 0x155A147B810C727F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   233 data: 0xC0407478 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29816  ts: 0x155A147B816C361F OLD: ts: 0x155A147B816C45BF dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   269 data: 0xC0487427 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29735  ts: 0x155A147B818C29CF OLD: ts: 0x155A147B818C413F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   417 data: 0xC048743B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29755  ts: 0x155A147B822BF40F OLD: ts: 0x155A147B822C0B7F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   441 data: 0xC040747C module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29820  ts: 0x155A147B824BE7BF OLD: ts: 0x155A147B824BF75F dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   623 data: 0xC048741E module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29726  ts: 0x155A147B830BA5AF OLD: ts: 0x155A147B830BBD1F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   653 data: 0xC040748B module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29835  ts: 0x155A147B832B995F OLD: ts: 0x155A147B832BB0CF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   677 data: 0xC0407477 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29815  ts: 0x155A147B834B94DF OLD: ts: 0x155A147B834BA47F dt: 0x0000000000000FA0 tag: 3 type: 0
 *** FEE64 discriminator: block:    48 ptr:   829 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B83EB4E99 OLD: ts: 0x155A147B83EB6005 dt: 0x000000000000116C tag: 2 type: 6
 ***            ADC data: block:    48 ptr:   839 data: 0xC0487432 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29746  ts: 0x155A147B83EB574F OLD: ts: 0x155A147B83EB6EBF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   917 data: 0xC0407482 module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29826  ts: 0x155A147B844B41FF OLD: ts: 0x155A147B844B4277 dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:   949 data: 0xC0607505 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29957  ts: 0x155A147B846B2DDF OLD: ts: 0x155A147B846B3A37 dt: 0x0000000000000C58 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  1047 data: 0xC0487429 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29737  ts: 0x155A147B84CB08EF OLD: ts: 0x155A147B84CB205F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1077 data: 0xC040748F module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29839  ts: 0x155A147B84EAFC9F OLD: ts: 0x155A147B84EB140F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1249 data: 0xC0487423 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29731  ts: 0x155A147B85AABA8F OLD: ts: 0x155A147B85AAD1FF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1307 data: 0xC0487431 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29745  ts: 0x155A147B85EAA1EF OLD: ts: 0x155A147B85EAB95F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1457 data: 0xC0487424 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29732  ts: 0x155A147B868A6C2F OLD: ts: 0x155A147B868A839F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1513 data: 0xC0487425 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29733  ts: 0x155A147B86CA538F OLD: ts: 0x155A147B86CA6AFF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1567 data: 0xC040747B module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29819  ts: 0x155A147B870A42BF OLD: ts: 0x155A147B870A48B9 dt: 0x00000000000005FA tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  1661 data: 0xC0487433 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29747  ts: 0x155A147B876A1DCF OLD: ts: 0x155A147B876A353F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1739 data: 0xC0607502 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29954  ts: 0x155A147B87C9F8DF OLD: ts: 0x155A147B87C9F96B dt: 0x000000000000008C tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  1861 data: 0xC0787515 module:  1 fail: 0 range: 0 id:   120 channel: 56 adc: 29973  ts: 0x155A147B8849CF6F OLD: ts: 0x155A147B8849E6DF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1945 data: 0xC0607503 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29955  ts: 0x155A147B88A9AA7F OLD: ts: 0x155A147B88A9AB97 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  2069 data: 0xC0607501 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29953  ts: 0x155A147B892988DF OLD: ts: 0x155A147B89298961 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  2077 data: 0xC048741B module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29723  ts: 0x155A147B8929810F OLD: ts: 0x155A147B8929987F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2137 data: 0xC048741C module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29724  ts: 0x155A147B8969686F OLD: ts: 0x155A147B89697FDF dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2219 data: 0xC0607505 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29957  ts: 0x155A147B89C9437F OLD: ts: 0x155A147B89C94401 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  2251 data: 0xC040748F module:  1 fail: 0 range: 0 id:    64 channel:  0 adc: 29839  ts: 0x155A147B89E93EFF OLD: ts: 0x155A147B89E94A03 dt: 0x0000000000000B04 tag: 2 type: 6
 *** FEE64 discriminator: block:    48 ptr:  2281 data: 0x81640001 module:  1 information type:  6 information field: 0x00040001 ts: 0x155A147B8A0927FB OLD: ts: 0x155A147B8A09398F dt: 0x0000000000001194 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  2349 data: 0xC0487428 module:  1 fail: 0 range: 0 id:    72 channel:  8 adc: 29736  ts: 0x155A147B8A491A0F OLD: ts: 0x155A147B8A49317F dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2485 data: 0xC06074F2 module:  1 fail: 0 range: 0 id:    96 channel: 32 adc: 29938  ts: 0x155A147B8AE8DC7F OLD: ts: 0x155A147B8AE8DCD9 dt: 0x000000000000005A tag: 2 type: 6
... 32812 more lines ...
Attachment 5: FEE1.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***            ADC data: block:    43 ptr:  1247 data: 0xC08875CE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30158  ts: 0x155A147B131355BD OLD: ts: 0x155A147B13136D2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1305 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B13533D1D OLD: ts: 0x155A147B1353548D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1513 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B1432EEBD OLD: ts: 0x155A147B1433062D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1571 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B1472D61D OLD: ts: 0x155A147B1472ED8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1627 data: 0xC08875AD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30125  ts: 0x155A147B14B2BD7D OLD: ts: 0x155A147B14B2D4ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1743 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B1532940D OLD: ts: 0x155A147B1532AB7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1801 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B15727B6D OLD: ts: 0x155A147B157292DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1861 data: 0xC08875B1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30129  ts: 0x155A147B15B262CD OLD: ts: 0x155A147B15B27A3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2011 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B16522D0D OLD: ts: 0x155A147B1652447D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2073 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B1692146D OLD: ts: 0x155A147B16922BDD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2273 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B1771C60D OLD: ts: 0x155A147B1771DD7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2339 data: 0xC08875CC module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30156  ts: 0x155A147B17B1AD6D OLD: ts: 0x155A147B17B1C4DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2545 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B18915F0D OLD: ts: 0x155A147B1891767D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2687 data: 0xC0907563 module:  2 fail: 0 range: 0 id:   144 channel: 16 adc: 30051  ts: 0x155A147B1931311D OLD: ts: 0x155A147B19313127 dt: 0x000000000000000A tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2697 data: 0xC08875BC module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30140  ts: 0x155A147B1931294D OLD: ts: 0x155A147B193140BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2751 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B197110AD OLD: ts: 0x155A147B1971281D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:    23 data: 0xC0807474 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29812  ts: 0x155A147B1991045D OLD: ts: 0x155A147B199113FD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:    59 data: 0xC08875CB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30155  ts: 0x155A147B19B0F80D OLD: ts: 0x155A147B19B10F7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   273 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B1A90A9AD OLD: ts: 0x155A147B1A90C11D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   335 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B1AD0910D OLD: ts: 0x155A147B1AD0A87D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   539 data: 0xC08875D0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30160  ts: 0x155A147B1BB042AD OLD: ts: 0x155A147B1BB05A1D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   593 data: 0xC0807466 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29798  ts: 0x155A147B1BF031DD OLD: ts: 0x155A147B1BF0417D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   745 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B1C8FF44D OLD: ts: 0x155A147B1C900BBD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1013 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B1DAF8D4D OLD: ts: 0x155A147B1DAFA4BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1225 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B1E8F3EED OLD: ts: 0x155A147B1E8F565D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1433 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B1F6EF08D OLD: ts: 0x155A147B1F6F07FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1693 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B208E898D OLD: ts: 0x155A147B208EA0FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1867 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B214E477D OLD: ts: 0x155A147B214E5EED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1927 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B218E2EDD OLD: ts: 0x155A147B218E464D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2075 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B222DF91D OLD: ts: 0x155A147B222E108D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2103 data: 0xC080748A module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29834  ts: 0x155A147B224DECCD OLD: ts: 0x155A147B224E043D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2133 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B226DE07D OLD: ts: 0x155A147B226DF7ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2197 data: 0xC08875C3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30147  ts: 0x155A147B22ADC7DD OLD: ts: 0x155A147B22ADDF4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2409 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B238D797D OLD: ts: 0x155A147B238D90ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2617 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B246D2B1D OLD: ts: 0x155A147B246D428D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2643 data: 0xC0807470 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29808  ts: 0x155A147B248D1ECD OLD: ts: 0x155A147B248D2E6D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2827 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B254CDCBD OLD: ts: 0x155A147B254CF42D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2891 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B258CC41D OLD: ts: 0x155A147B258CDB8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3039 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B262C8E5D OLD: ts: 0x155A147B262CA5CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3087 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B266C75BD OLD: ts: 0x155A147B266C8D2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3233 data: 0xC08875CA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30154  ts: 0x155A147B270C3FFD OLD: ts: 0x155A147B270C576D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3293 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B274C275D OLD: ts: 0x155A147B274C3ECD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3443 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B27EBF19D OLD: ts: 0x155A147B27EC090D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3507 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B282BD8FD OLD: ts: 0x155A147B282BF06D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3711 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B290B8A9D OLD: ts: 0x155A147B290BA20D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3857 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B29AB54DD OLD: ts: 0x155A147B29AB6C4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3973 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B2A2B239D OLD: ts: 0x155A147B2A2B3B0D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4245 data: 0xC08875CA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30154  ts: 0x155A147B2B4ABC9D OLD: ts: 0x155A147B2B4AD40D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4301 data: 0xC08875C2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30146  ts: 0x155A147B2B8AA3FD OLD: ts: 0x155A147B2B8ABB6D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4445 data: 0xC080747A module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29818  ts: 0x155A147B2C2A760D OLD: ts: 0x155A147B2C2A85AD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4539 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B2C8A511D OLD: ts: 0x155A147B2C8A688D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4595 data: 0xC08875AF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30127  ts: 0x155A147B2CCA387D OLD: ts: 0x155A147B2CCA4FED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4729 data: 0xC0A07467 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29799  ts: 0x155A147B2D6A02BD OLD: ts: 0x155A147B2D6A0321 dt: 0x0000000000000064 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4829 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B2DC9DDCD OLD: ts: 0x155A147B2DC9F53D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5031 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B2EA98F6D OLD: ts: 0x155A147B2EA9A6DD dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    44 ptr:  5331 data: 0x82620001 module:  2 information type:  6 information field: 0x00020001 ts: 0x155A147B200903F7 OLD: ts: 0x155A147B2FFB7FCF dt: 0x000000000FF27BD8 tag: 2 type: 4
 ***            ADC data: block:    44 ptr:  5349 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B20090FCD OLD: ts: 0x155A147B2009273D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5463 data: 0xC0807467 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29799  ts: 0x155A147B3088E65D OLD: ts: 0x155A147B3088F5FD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5619 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B3128A8CD OLD: ts: 0x155A147B3128C03D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:    69 data: 0xC0A07454 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29780  ts: 0x155A147B318883DD OLD: ts: 0x155A147B31888455 dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   253 data: 0xC08875CB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30155  ts: 0x155A147B324841CD OLD: ts: 0x155A147B3248593D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   313 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B3288292D OLD: ts: 0x155A147B3288409D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   369 data: 0xC08875CE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30158  ts: 0x155A147B32C8108D OLD: ts: 0x155A147B32C827FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   523 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B3367DACD OLD: ts: 0x155A147B3367F23D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   569 data: 0xC0A0745A module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29786  ts: 0x155A147B33A7C9FD OLD: ts: 0x155A147B33A7CA75 dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   577 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B33A7C22D OLD: ts: 0x155A147B33A7D99D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   783 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B348773CD OLD: ts: 0x155A147B34878B3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   839 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B34C75B2D OLD: ts: 0x155A147B34C7729D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   983 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147B3567256D OLD: ts: 0x155A147B35673CDD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1043 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B35A70CCD OLD: ts: 0x155A147B35A7243D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1103 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B35E6F42D OLD: ts: 0x155A147B35E70B9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1253 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B3686BE6D OLD: ts: 0x155A147B3686D5DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1313 data: 0xC08875B1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30129  ts: 0x155A147B36C6A5CD OLD: ts: 0x155A147B36C6BD3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1461 data: 0xC08875BC module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30140  ts: 0x155A147B3766700D OLD: ts: 0x155A147B3766877D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1721 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147B3886090D OLD: ts: 0x155A147B3886207D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1785 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B38C5F06D OLD: ts: 0x155A147B38C607DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1963 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B3985AE5D OLD: ts: 0x155A147B3985C5CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2027 data: 0xC08875D1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30161  ts: 0x155A147B39C595BD OLD: ts: 0x155A147B39C5AD2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2173 data: 0xC08875D1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30161  ts: 0x155A147B3A655FFD OLD: ts: 0x155A147B3A65776D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2203 data: 0xC0807464 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29796  ts: 0x155A147B3A8553AD OLD: ts: 0x155A147B3A856B1D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2383 data: 0xC08875D7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30167  ts: 0x155A147B3B45119D OLD: ts: 0x155A147B3B45290D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2595 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B3C24C33D OLD: ts: 0x155A147B3C24DAAD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2655 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B3C64AA9D OLD: ts: 0x155A147B3C64C20D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2801 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B3D0474DD OLD: ts: 0x155A147B3D048C4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2861 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B3D445C3D OLD: ts: 0x155A147B3D4473AD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3003 data: 0xC08875CE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30158  ts: 0x155A147B3DE4267D OLD: ts: 0x155A147B3DE43DED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3135 data: 0xC0A0744D module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29773  ts: 0x155A147B3E83F88D OLD: ts: 0x155A147B3E83F8B5 dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3143 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B3E83F0BD OLD: ts: 0x155A147B3E84082D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3201 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147B3EC3D81D OLD: ts: 0x155A147B3EC3EF8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3227 data: 0xC0807463 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29795  ts: 0x155A147B3EE3CBCD OLD: ts: 0x155A147B3EE3DB6D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3263 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B3F03BF7D OLD: ts: 0x155A147B3F03D6ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3465 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B3FE3711D OLD: ts: 0x155A147B3FE3888D dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    45 ptr:  3477 data: 0x82620001 module:  2 information type:  6 information field: 0x00020001 ts: 0x155A147B30035B27 OLD: ts: 0x155A147B3FF93FCF dt: 0x000000000FF5E4A8 tag: 2 type: 4
 ***            ADC data: block:    45 ptr:  3547 data: 0xC0A07460 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29792  ts: 0x155A147B40434C2D OLD: ts: 0x155A147B40434C69 dt: 0x000000000000003C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3663 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B40C322BD OLD: ts: 0x155A147B40C33A2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3881 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B41A2D45D OLD: ts: 0x155A147B41A2EBCD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3909 data: 0xC080746C module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29804  ts: 0x155A147B41C2C80D OLD: ts: 0x155A147B41C2DF7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3941 data: 0xC08875C7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30151  ts: 0x155A147B41E2BBBD OLD: ts: 0x155A147B41E2D32D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4001 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B4222A31D OLD: ts: 0x155A147B4222BA8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4139 data: 0xC0A07455 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29781  ts: 0x155A147B42C2752D OLD: ts: 0x155A147B42C2755F dt: 0x0000000000000032 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4149 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B42C26D5D OLD: ts: 0x155A147B42C284CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4171 data: 0xC0807468 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29800  ts: 0x155A147B42E2610D OLD: ts: 0x155A147B42E270AD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4229 data: 0xC0A07454 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29780  ts: 0x155A147B4322486D OLD: ts: 0x155A147B432248C7 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4297 data: 0xC08875B0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30128  ts: 0x155A147B4362379D OLD: ts: 0x155A147B43624F0D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4321 data: 0xC080747A module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29818  ts: 0x155A147B43822B4D OLD: ts: 0x155A147B43823AED dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4445 data: 0xC0A07456 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29782  ts: 0x155A147B440209AD OLD: ts: 0x155A147B440209DF dt: 0x0000000000000032 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4453 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B440201DD OLD: ts: 0x155A147B4402194D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4573 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B4481D09D OLD: ts: 0x155A147B4481E80D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4677 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B4501A72D OLD: ts: 0x155A147B4501BE9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4883 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B45E158CD OLD: ts: 0x155A147B45E1703D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4943 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B4621402D OLD: ts: 0x155A147B4621579D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5153 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B4700F1CD OLD: ts: 0x155A147B4701093D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5211 data: 0xC0B8745F module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29791  ts: 0x155A147B4740D92D OLD: ts: 0x155A147B4740F09D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5413 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B48208ACD OLD: ts: 0x155A147B4820A23D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5467 data: 0xC080747C module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29820  ts: 0x155A147B486079FD OLD: ts: 0x155A147B4860899D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5495 data: 0xC0A07450 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29776  ts: 0x155A147B488065DD OLD: ts: 0x155A147B488066A5 dt: 0x00000000000000C8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5623 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B49003C6D OLD: ts: 0x155A147B490053DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   135 data: 0xC08875AE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30126  ts: 0x155A147B49DFEE0D OLD: ts: 0x155A147B49E0057D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   157 data: 0xC0807472 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29810  ts: 0x155A147B49FFE1BD OLD: ts: 0x155A147B49FFF15D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   187 data: 0xC0807467 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29799  ts: 0x155A147B4A1FDD3D OLD: ts: 0x155A147B4A1FECDD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   395 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B4AFF870D OLD: ts: 0x155A147B4AFF9E7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   597 data: 0xC08875C3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30147  ts: 0x155A147B4BDF38AD OLD: ts: 0x155A147B4BDF501D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   655 data: 0xC0807469 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29801  ts: 0x155A147B4C1F27DD OLD: ts: 0x155A147B4C1F377D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   835 data: 0xC0B87444 module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29764  ts: 0x155A147B4CDED62D OLD: ts: 0x155A147B4CDEED9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1103 data: 0xC08875CA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30154  ts: 0x155A147B4DFE6F2D OLD: ts: 0x155A147B4DFE869D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1317 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B4EDE20CD OLD: ts: 0x155A147B4EDE383D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1605 data: 0xC0A07453 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29779  ts: 0x155A147B501DAD7D OLD: ts: 0x155A147B501DADFF dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1643 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B503DA12D OLD: ts: 0x155A147B503DB89D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1823 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B50FD5F1D OLD: ts: 0x155A147B50FD768D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1883 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B513D467D OLD: ts: 0x155A147B513D5DED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1941 data: 0xC08875C7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30151  ts: 0x155A147B517D2DDD OLD: ts: 0x155A147B517D454D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2145 data: 0xC08875C7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30151  ts: 0x155A147B525CDF7D OLD: ts: 0x155A147B525CF6ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2355 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B533C911D OLD: ts: 0x155A147B533CA88D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2469 data: 0xC0A07452 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29778  ts: 0x155A147B53BC67AD OLD: ts: 0x155A147B53BC6807 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2631 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B545C2A1D OLD: ts: 0x155A147B545C418D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2899 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B557BC31D OLD: ts: 0x155A147B557BDA8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3035 data: 0xC08875C2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30146  ts: 0x155A147B561B8D5D OLD: ts: 0x155A147B561BA4CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3093 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B565B74BD OLD: ts: 0x155A147B565B8C2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3243 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B56FB3EFD OLD: ts: 0x155A147B56FB566D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3301 data: 0xC0807463 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29795  ts: 0x155A147B573B2E2D OLD: ts: 0x155A147B573B3DCD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3573 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B585ABF5D OLD: ts: 0x155A147B585AD6CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3601 data: 0xC080746D module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29805  ts: 0x155A147B587AB30D OLD: ts: 0x155A147B587ACA7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3633 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B589AA6BD OLD: ts: 0x155A147B589ABE2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3773 data: 0xC0A07454 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29780  ts: 0x155A147B593A692D OLD: ts: 0x155A147B593A69D7 dt: 0x00000000000000AA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4021 data: 0xC08875AE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30126  ts: 0x155A147B5A3A0E7D OLD: ts: 0x155A147B5A3A25ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4077 data: 0xC08875AD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30125  ts: 0x155A147B5A79F5DD OLD: ts: 0x155A147B5A7A0D4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4183 data: 0xC0A0744F module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29775  ts: 0x155A147B5AF9CC6D OLD: ts: 0x155A147B5AF9CCB3 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4277 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B5B59A77D OLD: ts: 0x155A147B5B59BEED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4337 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B5B998EDD OLD: ts: 0x155A147B5B99A64D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4449 data: 0xC0A0744A module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29770  ts: 0x155A147B5C19656D OLD: ts: 0x155A147B5C196649 dt: 0x00000000000000DC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4509 data: 0xC080746A module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29802  ts: 0x155A147B5C594CCD OLD: ts: 0x155A147B5C595C6D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4545 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B5C79407D OLD: ts: 0x155A147B5C7957ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4667 data: 0xC08875A9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30121  ts: 0x155A147B5CF9170D OLD: ts: 0x155A147B5CF92E7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4931 data: 0xC08875CE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30158  ts: 0x155A147B5E18B00D OLD: ts: 0x155A147B5E18C77D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4991 data: 0xC08875C2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30146  ts: 0x155A147B5E58976D OLD: ts: 0x155A147B5E58AEDD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5261 data: 0xC08875AF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30127  ts: 0x155A147B5F78306D OLD: ts: 0x155A147B5F7847DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5403 data: 0xC0907565 module:  2 fail: 0 range: 0 id:   144 channel: 16 adc: 30053  ts: 0x155A147B6018027D OLD: ts: 0x155A147B60180287 dt: 0x000000000000000A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5413 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B6017FAAD OLD: ts: 0x155A147B6018121D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5467 data: 0xC08875CB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30155  ts: 0x155A147B6057E20D OLD: ts: 0x155A147B6057F97D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5617 data: 0xC08875AB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30123  ts: 0x155A147B60F7AC4D OLD: ts: 0x155A147B60F7C3BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   283 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B6277282D OLD: ts: 0x155A147B62773F9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   345 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B62B70F8D OLD: ts: 0x155A147B62B726FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   405 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B62F6F6ED OLD: ts: 0x155A147B62F70E5D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   485 data: 0xC0A0745E module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29790  ts: 0x155A147B6356D1FD OLD: ts: 0x155A147B6356D257 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   611 data: 0xC08875B1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30129  ts: 0x155A147B63D6A88D OLD: ts: 0x155A147B63D6BFFD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   761 data: 0xC08875D1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30161  ts: 0x155A147B647672CD OLD: ts: 0x155A147B64768A3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   821 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B64B65A2D OLD: ts: 0x155A147B64B6719D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1083 data: 0xC0A07456 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29782  ts: 0x155A147B65D5FAFD OLD: ts: 0x155A147B65D5FBCF dt: 0x00000000000000D2 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1091 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B65D5F32D OLD: ts: 0x155A147B65D60A9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1151 data: 0xC08875B2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30130  ts: 0x155A147B6615DA8D OLD: ts: 0x155A147B6615F1FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1359 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B66F58C2D OLD: ts: 0x155A147B66F5A39D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1619 data: 0xC08875A6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30118  ts: 0x155A147B6815252D OLD: ts: 0x155A147B68153C9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1727 data: 0xC0A07450 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29776  ts: 0x155A147B6894FBBD OLD: ts: 0x155A147B6894FC3F dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1791 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B68D4E31D OLD: ts: 0x155A147B68D4FA8D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1853 data: 0xC08875BC module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30140  ts: 0x155A147B6914CA7D OLD: ts: 0x155A147B6914E1ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2003 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B69B494BD OLD: ts: 0x155A147B69B4AC2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2149 data: 0xC08875C3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30147  ts: 0x155A147B6A545EFD OLD: ts: 0x155A147B6A54766D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2213 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B6A94465D OLD: ts: 0x155A147B6A945DCD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2273 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B6AD42DBD OLD: ts: 0x155A147B6AD4452D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2483 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B6BB3DF5D OLD: ts: 0x155A147B6BB3F6CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2543 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147B6BF3C6BD OLD: ts: 0x155A147B6BF3DE2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2565 data: 0xC0A07458 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29784  ts: 0x155A147B6C13BA6D OLD: ts: 0x155A147B6C13BB2B dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  2689 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B6C9390FD OLD: ts: 0x155A147B6C93A86D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  2957 data: 0xC08875D0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30160  ts: 0x155A147B6DB329FD OLD: ts: 0x155A147B6DB3416D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3151 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B6E92DB9D OLD: ts: 0x155A147B6E92F30D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3301 data: 0xC08875B2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30130  ts: 0x155A147B6F32A5DD OLD: ts: 0x155A147B6F32BD4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3361 data: 0xC0B87447 module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29767  ts: 0x155A147B6F728D3D OLD: ts: 0x155A147B6F72A4AD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3449 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B6FD2701D OLD: ts: 0x155A147B6FD2878D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3509 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147B7012577D OLD: ts: 0x155A147B70126EED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3661 data: 0xC08875AF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30127  ts: 0x155A147B70B221BD OLD: ts: 0x155A147B70B2392D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3719 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B70F2091D OLD: ts: 0x155A147B70F2208D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3743 data: 0xC0807469 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29801  ts: 0x155A147B7111FCCD OLD: ts: 0x155A147B71120C6D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3779 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B7131F07D OLD: ts: 0x155A147B713207ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3931 data: 0xC08875AD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30125  ts: 0x155A147B71D1BABD OLD: ts: 0x155A147B71D1D22D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  3991 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B7211A21D OLD: ts: 0x155A147B7211B98D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4139 data: 0xC08875C1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30145  ts: 0x155A147B72B16C5D OLD: ts: 0x155A147B72B183CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4203 data: 0xC08875CA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30154  ts: 0x155A147B72F153BD OLD: ts: 0x155A147B72F16B2D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4349 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B73911DFD OLD: ts: 0x155A147B7391356D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4567 data: 0xC08875CA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30154  ts: 0x155A147B7490C34D OLD: ts: 0x155A147B7490DABD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  4675 data: 0xC0A07458 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29784  ts: 0x155A147B751099DD OLD: ts: 0x155A147B75109A9B dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  4827 data: 0xC0A07464 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29796  ts: 0x155A147B75B0641D OLD: ts: 0x155A147B75B06459 dt: 0x000000000000003C tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  4837 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B75B05C4D OLD: ts: 0x155A147B75B073BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5041 data: 0xC08875B0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30128  ts: 0x155A147B76900DED OLD: ts: 0x155A147B7690255D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5099 data: 0xC0B8744A module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29770  ts: 0x155A147B76CFF54D OLD: ts: 0x155A147B76D00CBD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5303 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B77AFA6ED OLD: ts: 0x155A147B77AFBE5D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5577 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B78CF3FED OLD: ts: 0x155A147B78CF575D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  5635 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B790F274D OLD: ts: 0x155A147B790F3EBD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   147 data: 0xC0A0744F module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29775  ts: 0x155A147B79AEE9BD OLD: ts: 0x155A147B79AEE9EF dt: 0x0000000000000032 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:   273 data: 0xC08875AB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30123  ts: 0x155A147B7A2EC04D OLD: ts: 0x155A147B7A2ED7BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   325 data: 0xC08875BD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30141  ts: 0x155A147B7A6EA7AD OLD: ts: 0x155A147B7A6EBF1D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   479 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B7B0E71ED OLD: ts: 0x155A147B7B0E895D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   629 data: 0xC08875C0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30144  ts: 0x155A147B7BAE3C2D OLD: ts: 0x155A147B7BAE539D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:   831 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B7C8DEDCD OLD: ts: 0x155A147B7C8E053D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1041 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147B7D6D9F6D OLD: ts: 0x155A147B7D6DB6DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1253 data: 0xC08875C3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30147  ts: 0x155A147B7E4D510D OLD: ts: 0x155A147B7E4D687D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1401 data: 0xC08875AA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30122  ts: 0x155A147B7EED1B4D OLD: ts: 0x155A147B7EED32BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1461 data: 0xC08875B2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30130  ts: 0x155A147B7F2D02AD OLD: ts: 0x155A147B7F2D1A1D dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    48 ptr:  1643 data: 0x82610001 module:  2 information type:  6 information field: 0x00010001 ts: 0x155A147B700CBCF9 OLD: ts: 0x155A147B7FFFDFCF dt: 0x000000000FF322D6 tag: 2 type: 4
 ***            ADC data: block:    48 ptr:  1653 data: 0xC0A07469 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29801  ts: 0x155A147B700CBC1D OLD: ts: 0x155A147B700CBD03 dt: 0x00000000000000E6 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  1661 data: 0xC08875C7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30151  ts: 0x155A147B700CB44D OLD: ts: 0x155A147B700CCBBD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1717 data: 0xC08875D1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30161  ts: 0x155A147B804C9BAD OLD: ts: 0x155A147B804CB31D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1897 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B810C599D OLD: ts: 0x155A147B810C710D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  1949 data: 0xC0A07462 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29794  ts: 0x155A147B814C48CD OLD: ts: 0x155A147B814C4981 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  1957 data: 0xC08875C7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30151  ts: 0x155A147B814C40FD OLD: ts: 0x155A147B814C586D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2017 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B818C285D OLD: ts: 0x155A147B818C3FCD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2161 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B822BF29D OLD: ts: 0x155A147B822C0A0D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2365 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B830BA43D OLD: ts: 0x155A147B830BBBAD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2577 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B83EB55DD OLD: ts: 0x155A147B83EB6D4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  2785 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147B84CB077D OLD: ts: 0x155A147B84CB1EED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3053 data: 0xC0B8745F module:  2 fail: 0 range: 0 id:   184 channel: 56 adc: 29791  ts: 0x155A147B85EAA07D OLD: ts: 0x155A147B85EAB7ED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3209 data: 0xC08875AD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30125  ts: 0x155A147B868A6ABD OLD: ts: 0x155A147B868A822D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3269 data: 0xC08875AF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30127  ts: 0x155A147B86CA521D OLD: ts: 0x155A147B86CA698D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3415 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B876A1C5D OLD: ts: 0x155A147B876A33CD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3551 data: 0xC0A07465 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29797  ts: 0x155A147B8809EE6D OLD: ts: 0x155A147B8809EF2B dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  3559 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147B8809E69D OLD: ts: 0x155A147B8809FE0D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3827 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B89297F9D OLD: ts: 0x155A147B8929970D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3887 data: 0xC08875AE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30126  ts: 0x155A147B896966FD OLD: ts: 0x155A147B89697E6D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  3997 data: 0xC0A07467 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29799  ts: 0x155A147B89E93D8D OLD: ts: 0x155A147B89E93E41 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  4027 data: 0xC0A0745A module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29786  ts: 0x155A147B8A09390D OLD: ts: 0x155A147B8A093999 dt: 0x000000000000008C tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  4035 data: 0xC08875BE module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30142  ts: 0x155A147B8A09313D OLD: ts: 0x155A147B8A0948AD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4097 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B8A49189D OLD: ts: 0x155A147B8A49300D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4305 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B8B28CA3D OLD: ts: 0x155A147B8B28E1AD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4455 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B8BC8947D OLD: ts: 0x155A147B8BC8ABED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4517 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B8C087BDD OLD: ts: 0x155A147B8C08934D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4685 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147B8CC839CD OLD: ts: 0x155A147B8CC8513D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4745 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147B8D08212D OLD: ts: 0x155A147B8D08389D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  4955 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B8DE7D2CD OLD: ts: 0x155A147B8DE7EA3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  5107 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B8E879D0D OLD: ts: 0x155A147B8E87B47D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  5361 data: 0xC0A0746D module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29805  ts: 0x155A147B8FA73DDD OLD: ts: 0x155A147B8FA73E87 dt: 0x00000000000000AA tag: 2 type: 6
 ***            ADC data: block:    48 ptr:  5369 data: 0xC08875C5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30149  ts: 0x155A147B8FA7360D OLD: ts: 0x155A147B8FA74D7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  5425 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B8FE71D6D OLD: ts: 0x155A147B8FE734DD dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    48 ptr:  5437 data: 0x82610001 module:  2 information type:  6 information field: 0x00010001 ts: 0x155A147B80070EB1 OLD: ts: 0x155A147B8FFD9FCF dt: 0x000000000FF6911E tag: 2 type: 4
 ***            ADC data: block:    48 ptr:  5575 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B9086E7AD OLD: ts: 0x155A147B9086FF1D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    48 ptr:  5635 data: 0xC08875B1 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30129  ts: 0x155A147B90C6CF0D OLD: ts: 0x155A147B90C6E67D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:    89 data: 0xC08875C2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30146  ts: 0x155A147B9166994D OLD: ts: 0x155A147B9166B0BD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:   239 data: 0xC08875B6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30134  ts: 0x155A147B9206638D OLD: ts: 0x155A147B92067AFD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:   495 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147B9325FC8D OLD: ts: 0x155A147B932613FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:   733 data: 0xC0807476 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29814  ts: 0x155A147B9425A1DD OLD: ts: 0x155A147B9425B94D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:   763 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147B9445958D OLD: ts: 0x155A147B9445ACFD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:   909 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B94E55FCD OLD: ts: 0x155A147B94E5773D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1123 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B95C5116D OLD: ts: 0x155A147B95C528DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1543 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B978474AD OLD: ts: 0x155A147B97848C1D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1693 data: 0xC08875B5 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30133  ts: 0x155A147B98243EED OLD: ts: 0x155A147B9824565D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1811 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147B98A4157D OLD: ts: 0x155A147B98A42CED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1869 data: 0xC08875B4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30132  ts: 0x155A147B98E3FCDD OLD: ts: 0x155A147B98E4144D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  1929 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147B9923E43D OLD: ts: 0x155A147B9923FBAD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  2123 data: 0xC08875BA module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30138  ts: 0x155A147B9A0395DD OLD: ts: 0x155A147B9A03AD4D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  2261 data: 0xC0A0744B module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29771  ts: 0x155A147B9AA367ED OLD: ts: 0x155A147B9AA3688D dt: 0x00000000000000A0 tag: 2 type: 6
 ***            ADC data: block:    49 ptr:  2269 data: 0xC08875B0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30128  ts: 0x155A147B9AA3601D OLD: ts: 0x155A147B9AA3778D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  2329 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B9AE3477D OLD: ts: 0x155A147B9AE35EED dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  2539 data: 0xC08875C8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30152  ts: 0x155A147B9BC2F91D OLD: ts: 0x155A147B9BC3108D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  2775 data: 0xC0807479 module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29817  ts: 0x155A147B9CC29E6D OLD: ts: 0x155A147B9CC2AE0D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3049 data: 0xC080746B module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29803  ts: 0x155A147B9DE2376D OLD: ts: 0x155A147B9DE24EDD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3081 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147B9E022B1D OLD: ts: 0x155A147B9E02428D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3287 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B9EE1DCBD OLD: ts: 0x155A147B9EE1F42D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3441 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147B9F81A6FD OLD: ts: 0x155A147B9F81BE6D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3501 data: 0xC08875C2 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30146  ts: 0x155A147B9FC18E5D OLD: ts: 0x155A147B9FC1A5CD dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    49 ptr:  3543 data: 0x82620001 module:  2 information type:  6 information field: 0x00020001 ts: 0x155A147B90016E83 OLD: ts: 0x155A147B9FFB5FCF dt: 0x000000000FF9F14C tag: 2 type: 4
 ***            ADC data: block:    49 ptr:  3555 data: 0xC080746A module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29802  ts: 0x155A147B90017D8D OLD: ts: 0x155A147B90018D2D dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3645 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147BA061589D OLD: ts: 0x155A147BA061700D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3705 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147BA0A13FFD OLD: ts: 0x155A147BA0A1576D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3757 data: 0xC080747E module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29822  ts: 0x155A147BA0E12F2D OLD: ts: 0x155A147BA0E13ECD dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3913 data: 0xC08875C9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30153  ts: 0x155A147BA180F19D OLD: ts: 0x155A147BA181090D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  3977 data: 0xC08875B7 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30135  ts: 0x155A147BA1C0D8FD OLD: ts: 0x155A147BA1C0F06D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4243 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147BA2E071FD OLD: ts: 0x155A147BA2E0896D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4381 data: 0xC0A07457 module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29783  ts: 0x155A147BA380440D OLD: ts: 0x155A147BA38044C1 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    49 ptr:  4389 data: 0xC08875C4 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30148  ts: 0x155A147BA3803C3D OLD: ts: 0x155A147BA38053AD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4421 data: 0xC080746B module:  2 fail: 0 range: 0 id:   128 channel:  0 adc: 29803  ts: 0x155A147BA3A02FED OLD: ts: 0x155A147BA3A0475D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4451 data: 0xC08875C6 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30150  ts: 0x155A147BA3C0239D OLD: ts: 0x155A147BA3C03B0D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4627 data: 0xC08875BF module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30143  ts: 0x155A147BA47FE18D OLD: ts: 0x155A147BA47FF8FD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4685 data: 0xC08875CB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30155  ts: 0x155A147BA4BFC8ED OLD: ts: 0x155A147BA4BFE05D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  4831 data: 0xC08875B3 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30131  ts: 0x155A147BA55F932D OLD: ts: 0x155A147BA55FAA9D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  5033 data: 0xC08875B9 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30137  ts: 0x155A147BA63F44CD OLD: ts: 0x155A147BA63F5C3D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  5241 data: 0xC08875BB module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30139  ts: 0x155A147BA71EF66D OLD: ts: 0x155A147BA71F0DDD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  5395 data: 0xC08875B8 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30136  ts: 0x155A147BA7BEC0AD OLD: ts: 0x155A147BA7BED81D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  5455 data: 0xC08875B0 module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30128  ts: 0x155A147BA7FEA80D OLD: ts: 0x155A147BA7FEBF7D dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    49 ptr:  5515 data: 0xC08875BC module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30140  ts: 0x155A147BA83E8F6D OLD: ts: 0x155A147BA83EA6DD dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    50 ptr:    21 data: 0xC0A0744B module:  2 fail: 0 range: 0 id:   160 channel: 32 adc: 29771  ts: 0x155A147BA91E48DD OLD: ts: 0x155A147BA91E494B dt: 0x000000000000006E tag: 2 type: 6
 ***            ADC data: block:    50 ptr:    29 data: 0xC08875AD module:  2 fail: 0 range: 0 id:   136 channel:  8 adc: 30125  ts: 0x155A147BA91E410D OLD: ts: 0x155A147BA91E587D dt: 0x0000000000001770 tag: 3 type: 0
... 26630 more lines ...
Attachment 6: FEE2.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 ***            ADC data: block:    43 ptr:  1025 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B1432EF12 OLD: ts: 0x155A147B14330682 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1083 data: 0xC0C8749C module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29852  ts: 0x155A147B1472D672 OLD: ts: 0x155A147B1472EDE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1143 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B14B2BDD2 OLD: ts: 0x155A147B14B2D542 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1291 data: 0xC0C87497 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29847  ts: 0x155A147B15528812 OLD: ts: 0x155A147B15529F82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1351 data: 0xC0C874AD module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29869  ts: 0x155A147B15926F72 OLD: ts: 0x155A147B159286E2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1411 data: 0xC0C874A6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29862  ts: 0x155A147B15D256D2 OLD: ts: 0x155A147B15D26E42 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1433 data: 0xC0C07484 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29828  ts: 0x155A147B15F24A82 OLD: ts: 0x155A147B15F24B7C dt: 0x00000000000000FA tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1561 data: 0xC0C874B4 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29876  ts: 0x155A147B16722112 OLD: ts: 0x155A147B16723882 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1623 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B16B20872 OLD: ts: 0x155A147B16B21FE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1653 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B16D1FC22 OLD: ts: 0x155A147B16D21392 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1677 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B16F1EFD2 OLD: ts: 0x155A147B16F1F234 dt: 0x0000000000000262 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1707 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B1711EB52 OLD: ts: 0x155A147B1711EC4C dt: 0x00000000000000FA tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1741 data: 0xC0C0747B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29819  ts: 0x155A147B1731E6D2 OLD: ts: 0x155A147B1731F000 dt: 0x000000000000092E tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1809 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B1771C662 OLD: ts: 0x155A147B1771DDD2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1869 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B17B1ADC2 OLD: ts: 0x155A147B17B1C532 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1929 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B17F19522 OLD: ts: 0x155A147B17F1AC92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1949 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B181188D2 OLD: ts: 0x155A147B18118990 dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1979 data: 0xC0C0747B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29819  ts: 0x155A147B18318452 OLD: ts: 0x155A147B183184DE dt: 0x000000000000008C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2077 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B18915F62 OLD: ts: 0x155A147B189176D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2137 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B18D146C2 OLD: ts: 0x155A147B18D15E32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2189 data: 0xC0D0750E module:  3 fail: 0 range: 0 id:   208 channel: 16 adc: 29966  ts: 0x155A147B191135F2 OLD: ts: 0x155A147B19113B1A dt: 0x0000000000000528 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2289 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B19711102 OLD: ts: 0x155A147B19712872 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2349 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B19B0F862 OLD: ts: 0x155A147B19B10FD2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2435 data: 0xC0C0747E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29822  ts: 0x155A147B1A10D372 OLD: ts: 0x155A147B1A10D4E4 dt: 0x0000000000000172 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2523 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B1A70B652 OLD: ts: 0x155A147B1A70B7BA dt: 0x0000000000000168 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2553 data: 0xC0C07496 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29846  ts: 0x155A147B1A90B1D2 OLD: ts: 0x155A147B1A90B3C6 dt: 0x00000000000001F4 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2623 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B1AD09162 OLD: ts: 0x155A147B1AD0A8D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2683 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B1B1078C2 OLD: ts: 0x155A147B1B109032 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2705 data: 0xC0C07487 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29831  ts: 0x155A147B1B306C72 OLD: ts: 0x155A147B1B306D30 dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2833 data: 0xC0C874A6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29862  ts: 0x155A147B1BB04302 OLD: ts: 0x155A147B1BB05A72 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2893 data: 0xC0C87492 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29842  ts: 0x155A147B1BF02A62 OLD: ts: 0x155A147B1BF041D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3003 data: 0xC0C07496 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29846  ts: 0x155A147B1C7000F2 OLD: ts: 0x155A147B1C7004EE dt: 0x00000000000003FC tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3045 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B1C8FF4A2 OLD: ts: 0x155A147B1C900C12 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3103 data: 0xC0C874BC module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29884  ts: 0x155A147B1CCFDC02 OLD: ts: 0x155A147B1CCFF372 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3311 data: 0xC0C8749C module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29852  ts: 0x155A147B1DAF8DA2 OLD: ts: 0x155A147B1DAFA512 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3393 data: 0xC0C87497 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29847  ts: 0x155A147B1E0F68B2 OLD: ts: 0x155A147B1E0F6DBC dt: 0x000000000000050A tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3513 data: 0xC0C07497 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29847  ts: 0x155A147B1E8F4712 OLD: ts: 0x155A147B1E8F49EC dt: 0x00000000000002DA tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3583 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B1ECF26A2 OLD: ts: 0x155A147B1ECF3E12 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:    21 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B1EEF1A52 OLD: ts: 0x155A147B1EEF1B06 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   151 data: 0xC0C874A9 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29865  ts: 0x155A147B1F6EF0E2 OLD: ts: 0x155A147B1F6F0852 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   209 data: 0xC0F07585 module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30085  ts: 0x155A147B1FAEE012 OLD: ts: 0x155A147B1FAEEFB2 dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   323 data: 0xC0C0749A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29850  ts: 0x155A147B202EAED2 OLD: ts: 0x155A147B202EAF9A dt: 0x00000000000000C8 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   421 data: 0xC0C874A7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29863  ts: 0x155A147B208E89E2 OLD: ts: 0x155A147B208EA152 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   481 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B20CE7142 OLD: ts: 0x155A147B20CE88B2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   631 data: 0xC0C874A9 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29865  ts: 0x155A147B216E3B82 OLD: ts: 0x155A147B216E52F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   713 data: 0xC0C07478 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29816  ts: 0x155A147B21CE1692 OLD: ts: 0x155A147B21CE1AE8 dt: 0x0000000000000456 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   743 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B21EE1212 OLD: ts: 0x155A147B21EE14E2 dt: 0x00000000000002D0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   843 data: 0xC0C874AC module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29868  ts: 0x155A147B224DED22 OLD: ts: 0x155A147B224E0492 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   903 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B228DD482 OLD: ts: 0x155A147B228DEBF2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   931 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B22ADC832 OLD: ts: 0x155A147B22ADD7D2 dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   993 data: 0xC0C874AF module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29871  ts: 0x155A147B22EDAF92 OLD: ts: 0x155A147B22EDC702 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1015 data: 0xC0C07488 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29832  ts: 0x155A147B230DA342 OLD: ts: 0x155A147B230DA626 dt: 0x00000000000002E4 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1143 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B238D79D2 OLD: ts: 0x155A147B238D9142 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1343 data: 0xC0C07482 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29826  ts: 0x155A147B246D3342 OLD: ts: 0x155A147B246D35E0 dt: 0x000000000000029E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1411 data: 0xC0C874A4 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29860  ts: 0x155A147B24AD12D2 OLD: ts: 0x155A147B24AD2A42 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1565 data: 0xC0C874A3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29859  ts: 0x155A147B254CDD12 OLD: ts: 0x155A147B254CF482 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1623 data: 0xC0C874A3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29859  ts: 0x155A147B258CC472 OLD: ts: 0x155A147B258CDBE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1831 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B266C7612 OLD: ts: 0x155A147B266C8D82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1961 data: 0xC0C07495 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29845  ts: 0x155A147B270C4822 OLD: ts: 0x155A147B270C48E0 dt: 0x00000000000000BE tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2027 data: 0xC0C87499 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29849  ts: 0x155A147B274C27B2 OLD: ts: 0x155A147B274C3F22 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2143 data: 0xC0C07472 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29810  ts: 0x155A147B27CBFE42 OLD: ts: 0x155A147B27CC01F8 dt: 0x00000000000003B6 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2173 data: 0xC0C0748F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29839  ts: 0x155A147B27EBF9C2 OLD: ts: 0x155A147B27EBFB3E dt: 0x000000000000017C tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2241 data: 0xC0C87499 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29849  ts: 0x155A147B282BD952 OLD: ts: 0x155A147B282BF0C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2353 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B28ABAFE2 OLD: ts: 0x155A147B28ABB4E2 dt: 0x0000000000000500 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2439 data: 0xC0C0748B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29835  ts: 0x155A147B290B92C2 OLD: ts: 0x155A147B290B9560 dt: 0x000000000000029E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2507 data: 0xC0C874AD module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29869  ts: 0x155A147B294B7252 OLD: ts: 0x155A147B294B89C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2557 data: 0xC0C07494 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29844  ts: 0x155A147B298B6182 OLD: ts: 0x155A147B298B6326 dt: 0x00000000000001A4 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2587 data: 0xC0C0747B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29819  ts: 0x155A147B29AB5D02 OLD: ts: 0x155A147B29AB5F3C dt: 0x000000000000023A tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2713 data: 0xC0C87499 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29849  ts: 0x155A147B2A2B23F2 OLD: ts: 0x155A147B2A2B3B62 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2779 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B2A6B0B52 OLD: ts: 0x155A147B2A6B22C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2861 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x155A147B2ACAE662 OLD: ts: 0x155A147B2ACAE7D4 dt: 0x0000000000000172 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2987 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B2B4ABCF2 OLD: ts: 0x155A147B2B4AD462 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3047 data: 0xC0C874A8 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29864  ts: 0x155A147B2B8AA452 OLD: ts: 0x155A147B2B8ABBC2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3191 data: 0xC0C874B4 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29876  ts: 0x155A147B2C2A6E92 OLD: ts: 0x155A147B2C2A8602 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3213 data: 0xC0C0747E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29822  ts: 0x155A147B2C4A6242 OLD: ts: 0x155A147B2C4A6512 dt: 0x00000000000002D0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3241 data: 0xC0C07488 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29832  ts: 0x155A147B2C6A5DC2 OLD: ts: 0x155A147B2C6A6150 dt: 0x000000000000038E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3271 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B2C8A5942 OLD: ts: 0x155A147B2C8A5BCC dt: 0x000000000000028A tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3339 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B2CCA38D2 OLD: ts: 0x155A147B2CCA5042 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3423 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B2D2A13E2 OLD: ts: 0x155A147B2D2A1720 dt: 0x000000000000033E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3601 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B2DE9D1D2 OLD: ts: 0x155A147B2DE9E942 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3659 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B2E29B932 OLD: ts: 0x155A147B2E29D0A2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3679 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B2E49ACE2 OLD: ts: 0x155A147B2E49ADD2 dt: 0x00000000000000F0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3837 data: 0xC0C874A0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29856  ts: 0x155A147B2EE97722 OLD: ts: 0x155A147B2EE98E92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3891 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B2F296652 OLD: ts: 0x155A147B2F297642 dt: 0x0000000000000FF0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4115 data: 0xC0C87498 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29848  ts: 0x155A147B30091022 OLD: ts: 0x155A147B30092792 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4199 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B3068EB32 OLD: ts: 0x155A147B3068F398 dt: 0x0000000000000866 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4237 data: 0xC0C87497 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29847  ts: 0x155A147B3088DEE2 OLD: ts: 0x155A147B3088F652 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4259 data: 0xC0C07496 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29846  ts: 0x155A147B30A8D292 OLD: ts: 0x155A147B30A8D490 dt: 0x00000000000001FE tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4377 data: 0xC0C0748B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29835  ts: 0x155A147B3128B0F2 OLD: ts: 0x155A147B3128B48A dt: 0x0000000000000398 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4441 data: 0xC0C874A7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29863  ts: 0x155A147B31689082 OLD: ts: 0x155A147B3168A7F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4463 data: 0xC0C07484 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29828  ts: 0x155A147B31888432 OLD: ts: 0x155A147B31888450 dt: 0x000000000000001E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4581 data: 0xC0C07493 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29843  ts: 0x155A147B32086292 OLD: ts: 0x155A147B32086788 dt: 0x00000000000004F6 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4611 data: 0xC0C0747E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29822  ts: 0x155A147B32284E72 OLD: ts: 0x155A147B322852BE dt: 0x000000000000044C tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4641 data: 0xC0C07495 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29845  ts: 0x155A147B324849F2 OLD: ts: 0x155A147B32484C2C dt: 0x000000000000023A tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4701 data: 0xC0C07474 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29812  ts: 0x155A147B32883152 OLD: ts: 0x155A147B328834EA dt: 0x0000000000000398 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4771 data: 0xC0C874AD module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29869  ts: 0x155A147B32C810E2 OLD: ts: 0x155A147B32C82852 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4973 data: 0xC0C0748D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29837  ts: 0x155A147B33A7CA52 OLD: ts: 0x155A147B33A7CA70 dt: 0x000000000000001E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5041 data: 0xC0C874A7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29863  ts: 0x155A147B33E7A9E2 OLD: ts: 0x155A147B33E7C152 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5063 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B34079D92 OLD: ts: 0x155A147B3407A1CA dt: 0x0000000000000438 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5187 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B34877422 OLD: ts: 0x155A147B34878B92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5245 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B34C75B82 OLD: ts: 0x155A147B34C772F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5357 data: 0xC0C0748D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29837  ts: 0x155A147B35473212 OLD: ts: 0x155A147B354736CC dt: 0x00000000000004BA tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5391 data: 0xC0C07490 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29840  ts: 0x155A147B35672D92 OLD: ts: 0x155A147B35672ED2 dt: 0x0000000000000140 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5451 data: 0xC0C07488 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29832  ts: 0x155A147B35A714F2 OLD: ts: 0x155A147B35A71830 dt: 0x000000000000033E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5519 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B35E6F482 OLD: ts: 0x155A147B35E70BF2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  5633 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B3666CB12 OLD: ts: 0x155A147B3666CF4A dt: 0x0000000000000438 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5663 data: 0xC0C07495 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29845  ts: 0x155A147B3686C692 OLD: ts: 0x155A147B3686C8EA dt: 0x0000000000000258 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5731 data: 0xC0C87497 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29847  ts: 0x155A147B36C6A622 OLD: ts: 0x155A147B36C6BD92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   117 data: 0xC0C874A7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29863  ts: 0x155A147B37667062 OLD: ts: 0x155A147B376687D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   173 data: 0xC0C87497 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29847  ts: 0x155A147B37A657C2 OLD: ts: 0x155A147B37A66F32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   255 data: 0xC0C07474 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29812  ts: 0x155A147B380632D2 OLD: ts: 0x155A147B380636E2 dt: 0x0000000000000410 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   289 data: 0xC0C07495 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29845  ts: 0x155A147B38262E52 OLD: ts: 0x155A147B3826301E dt: 0x00000000000001CC tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   379 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B38861132 OLD: ts: 0x155A147B388611E6 dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   447 data: 0xC0C87493 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29843  ts: 0x155A147B38C5F0C2 OLD: ts: 0x155A147B38C60832 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   507 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B3905D822 OLD: ts: 0x155A147B3905EF92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   559 data: 0xC0C07472 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29810  ts: 0x155A147B3945C752 OLD: ts: 0x155A147B3945C98C dt: 0x000000000000023A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   655 data: 0xC0C87496 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29846  ts: 0x155A147B39A5A262 OLD: ts: 0x155A147B39A5B9D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   715 data: 0xC0C87493 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29843  ts: 0x155A147B39E589C2 OLD: ts: 0x155A147B39E5A132 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   863 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B3A855402 OLD: ts: 0x155A147B3A856B72 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   893 data: 0xC0C874B6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29878  ts: 0x155A147B3AA547B2 OLD: ts: 0x155A147B3AA55F22 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   915 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B3AC53B62 OLD: ts: 0x155A147B3AC53CFC dt: 0x000000000000019A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   951 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B3AE536E2 OLD: ts: 0x155A147B3AE54006 dt: 0x0000000000000924 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1049 data: 0xC0C874B3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29875  ts: 0x155A147B3B4511F2 OLD: ts: 0x155A147B3B452962 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1079 data: 0xC0F07581 module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30081  ts: 0x155A147B3B6505A2 OLD: ts: 0x155A147B3B651542 dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1131 data: 0xC0C0748B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29835  ts: 0x155A147B3BA4ED02 OLD: ts: 0x155A147B3BA4EF32 dt: 0x0000000000000230 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1221 data: 0xC0C07494 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29844  ts: 0x155A147B3C04CFE2 OLD: ts: 0x155A147B3C04D500 dt: 0x000000000000051E tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1253 data: 0xC0C07479 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29817  ts: 0x155A147B3C24CB62 OLD: ts: 0x155A147B3C24CF5E dt: 0x00000000000003FC tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1321 data: 0xC0C874A7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29863  ts: 0x155A147B3C64AAF2 OLD: ts: 0x155A147B3C64C262 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1471 data: 0xC0C87493 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29843  ts: 0x155A147B3D047532 OLD: ts: 0x155A147B3D048CA2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1531 data: 0xC0C874A6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29862  ts: 0x155A147B3D445C92 OLD: ts: 0x155A147B3D447402 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1583 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B3D844BC2 OLD: ts: 0x155A147B3D844D3E dt: 0x000000000000017C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1683 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B3DE426D2 OLD: ts: 0x155A147B3DE43E42 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1765 data: 0xC0C07499 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29849  ts: 0x155A147B3E4401E2 OLD: ts: 0x155A147B3E440638 dt: 0x0000000000000456 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1797 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B3E63FD62 OLD: ts: 0x155A147B3E640E24 dt: 0x00000000000010C2 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1883 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B3EC3D872 OLD: ts: 0x155A147B3EC3EFE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1943 data: 0xC0C874AC module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29868  ts: 0x155A147B3F03BFD2 OLD: ts: 0x155A147B3F03D742 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2151 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B3FE37172 OLD: ts: 0x155A147B3FE388E2 dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    45 ptr:  2163 data: 0x83620001 module:  3 information type:  6 information field: 0x00020001 ts: 0x155A147B30035B22 OLD: ts: 0x155A147B3FF5DCA4 dt: 0x000000000FF28182 tag: 2 type: 4
 ***            ADC data: block:    45 ptr:  2211 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B402358D2 OLD: ts: 0x155A147B40237042 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2365 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B40C32312 OLD: ts: 0x155A147B40C33A82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2393 data: 0xC0F07585 module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30085  ts: 0x155A147B40E316C2 OLD: ts: 0x155A147B40E32662 dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2477 data: 0xC0C07479 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29817  ts: 0x155A147B4142F9A2 OLD: ts: 0x155A147B4142FD8A dt: 0x00000000000003E8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2509 data: 0xC0C07486 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29830  ts: 0x155A147B4162F522 OLD: ts: 0x155A147B41630012 dt: 0x0000000000000AF0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2577 data: 0xC0C87498 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29848  ts: 0x155A147B41A2D4B2 OLD: ts: 0x155A147B41A2EC22 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2635 data: 0xC0C874AD module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29869  ts: 0x155A147B41E2BC12 OLD: ts: 0x155A147B41E2D382 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2695 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B4222A372 OLD: ts: 0x155A147B4222BAE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2903 data: 0xC0C87493 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29843  ts: 0x155A147B43025512 OLD: ts: 0x155A147B43026C82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2959 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B43424442 OLD: ts: 0x155A147B43424758 dt: 0x0000000000000316 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2989 data: 0xC0C07474 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29812  ts: 0x155A147B43623FC2 OLD: ts: 0x155A147B436241A2 dt: 0x00000000000001E0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3055 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B43A21F52 OLD: ts: 0x155A147B43A236C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3077 data: 0xC0C07490 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29840  ts: 0x155A147B43C21302 OLD: ts: 0x155A147B43C214EC dt: 0x00000000000001EA tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3107 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x155A147B43E20E82 OLD: ts: 0x155A147B43E2118E dt: 0x000000000000030C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3267 data: 0xC0C874B1 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29873  ts: 0x155A147B4481D0F2 OLD: ts: 0x155A147B4481E862 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3463 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B45618292 OLD: ts: 0x155A147B45619A02 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3485 data: 0xC0C0748D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29837  ts: 0x155A147B45817642 OLD: ts: 0x155A147B458178E0 dt: 0x000000000000029E tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  3545 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B45C16112 OLD: ts: 0x155A147B45C16D42 dt: 0x0000000000000C30 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3681 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B46413432 OLD: ts: 0x155A147B46414BA2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3821 data: 0xC0C07484 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29828  ts: 0x155A147B46E0FE72 OLD: ts: 0x155A147B46E10322 dt: 0x00000000000004B0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3919 data: 0xC0C874B4 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29876  ts: 0x155A147B4740D982 OLD: ts: 0x155A147B4740F0F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4033 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x155A147B47C0B012 OLD: ts: 0x155A147B47C0B904 dt: 0x00000000000008F2 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4127 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B48208B22 OLD: ts: 0x155A147B4820A292 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4187 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B48607282 OLD: ts: 0x155A147B486089F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4209 data: 0xC0C0747D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29821  ts: 0x155A147B48806632 OLD: ts: 0x155A147B488066AA dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4341 data: 0xC0C87494 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29844  ts: 0x155A147B49003CC2 OLD: ts: 0x155A147B49005432 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4423 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B496017D2 OLD: ts: 0x155A147B49601CA0 dt: 0x00000000000004CE tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4543 data: 0xC0C07483 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29827  ts: 0x155A147B49DFF632 OLD: ts: 0x155A147B49DFF858 dt: 0x0000000000000226 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4613 data: 0xC0C87496 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29846  ts: 0x155A147B4A1FD5C2 OLD: ts: 0x155A147B4A1FED32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4695 data: 0xC0C07493 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29843  ts: 0x155A147B4A7FB0D2 OLD: ts: 0x155A147B4A7FB56E dt: 0x000000000000049C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4823 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B4AFF8762 OLD: ts: 0x155A147B4AFF9ED2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4879 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B4B3F7692 OLD: ts: 0x155A147B4B3F7FA2 dt: 0x0000000000000910 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4909 data: 0xC0C07493 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29843  ts: 0x155A147B4B5F6272 OLD: ts: 0x155A147B4B5F6722 dt: 0x00000000000004B0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4943 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B4B7F5DF2 OLD: ts: 0x155A147B4B7F614E dt: 0x000000000000035C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5039 data: 0xC0C874AD module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29869  ts: 0x155A147B4BDF3902 OLD: ts: 0x155A147B4BDF5072 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5099 data: 0xC0C874A1 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29857  ts: 0x155A147B4C1F2062 OLD: ts: 0x155A147B4C1F37D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5157 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B4C5F07C2 OLD: ts: 0x155A147B4C5F1F32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5217 data: 0xC0C874B3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29875  ts: 0x155A147B4C9EEF22 OLD: ts: 0x155A147B4C9F0692 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5239 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B4CBEE2D2 OLD: ts: 0x155A147B4CBEE3A4 dt: 0x00000000000000D2 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5281 data: 0xC0C87494 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29844  ts: 0x155A147B4CDED682 OLD: ts: 0x155A147B4CDEEDF2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5361 data: 0xC0C07488 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29832  ts: 0x155A147B4D3EB192 OLD: ts: 0x155A147B4D3EB3F4 dt: 0x0000000000000262 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5391 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B4D5EAD12 OLD: ts: 0x155A147B4D5EAF92 dt: 0x0000000000000280 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5423 data: 0xC0C07479 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29817  ts: 0x155A147B4D7EA892 OLD: ts: 0x155A147B4D7EB152 dt: 0x00000000000008C0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5453 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B4D9E9472 OLD: ts: 0x155A147B4D9E9562 dt: 0x00000000000000F0 tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  5483 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B4DBE8084 OLD: ts: 0x155A147B4DBE8F0C dt: 0x0000000000000E88 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5553 data: 0xC0C874AE module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29870  ts: 0x155A147B4DFE6F82 OLD: ts: 0x155A147B4DFE86F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5617 data: 0xC0C874AF module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29871  ts: 0x155A147B4E3E56E2 OLD: ts: 0x155A147B4E3E6E52 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5639 data: 0xC0C07475 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29813  ts: 0x155A147B4E5E4A92 OLD: ts: 0x155A147B4E5E4DDA dt: 0x0000000000000348 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  5669 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B4E7E4612 OLD: ts: 0x155A147B4E7E47F2 dt: 0x00000000000001E0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:     7 data: 0xC0C874AE module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29870  ts: 0x155A147B4EDE2122 OLD: ts: 0x155A147B4EDE3892 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:    65 data: 0xC0F0758B module:  3 fail: 0 range: 0 id:   240 channel: 48 adc: 30091  ts: 0x155A147B4F1E1052 OLD: ts: 0x155A147B4F1E1FF2 dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   177 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B4F9DDF12 OLD: ts: 0x155A147B4F9DE084 dt: 0x0000000000000172 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   237 data: 0xC0C07493 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29843  ts: 0x155A147B4FDDC672 OLD: ts: 0x155A147B4FDDC834 dt: 0x00000000000001C2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   297 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B501DADD2 OLD: ts: 0x155A147B501DADFA dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   327 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B503DA952 OLD: ts: 0x155A147B503DABD2 dt: 0x0000000000000280 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   397 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B507D88E2 OLD: ts: 0x155A147B507DA052 dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    46 ptr:   513 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B50FD5A68 OLD: ts: 0x155A147B50FD680A dt: 0x0000000000000DA2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   553 data: 0xC0C8749E module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29854  ts: 0x155A147B511D5322 OLD: ts: 0x155A147B511D6A92 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   615 data: 0xC0C87496 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29846  ts: 0x155A147B515D3A82 OLD: ts: 0x155A147B515D51F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   673 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B519D21E2 OLD: ts: 0x155A147B519D3952 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   697 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B51BD1592 OLD: ts: 0x155A147B51BD1678 dt: 0x00000000000000E6 tag: 2 type: 6
 *** FEE64 discriminator: block:    46 ptr:   787 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B521CF07A OLD: ts: 0x155A147B521CFE3A dt: 0x0000000000000DC0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   917 data: 0xC0C874AF module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29871  ts: 0x155A147B529CC732 OLD: ts: 0x155A147B529CDEA2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   939 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x155A147B52BCBAE2 OLD: ts: 0x155A147B52BCBBC8 dt: 0x00000000000000E6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   969 data: 0xC0C07497 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29847  ts: 0x155A147B52DCB662 OLD: ts: 0x155A147B52DCB84C dt: 0x00000000000001EA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1067 data: 0xC0C87496 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29846  ts: 0x155A147B533C9172 OLD: ts: 0x155A147B533CA8E2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1149 data: 0xC0C0748D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29837  ts: 0x155A147B539C6C82 OLD: ts: 0x155A147B539C713C dt: 0x00000000000004BA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1333 data: 0xC0C07494 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29844  ts: 0x155A147B545C3242 OLD: ts: 0x155A147B545C354E dt: 0x000000000000030C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1401 data: 0xC0C874B0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29872  ts: 0x155A147B549C11D2 OLD: ts: 0x155A147B549C2942 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1481 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B54FBECE2 OLD: ts: 0x155A147B54FBF1CE dt: 0x00000000000004EC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1607 data: 0xC0C87498 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29848  ts: 0x155A147B557BC372 OLD: ts: 0x155A147B557BDAE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1667 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B55BBAAD2 OLD: ts: 0x155A147B55BBC242 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1689 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B55DB9E82 OLD: ts: 0x155A147B55DBA1DE dt: 0x000000000000035C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1749 data: 0xC0C07494 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29844  ts: 0x155A147B561B9582 OLD: ts: 0x155A147B561B983E dt: 0x00000000000002BC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1869 data: 0xC0C8749C module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29852  ts: 0x155A147B569B5C72 OLD: ts: 0x155A147B569B73E2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1891 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B56BB5022 OLD: ts: 0x155A147B56BB552C dt: 0x000000000000050A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1957 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B56FB3F52 OLD: ts: 0x155A147B56FB56C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2017 data: 0xC0C874A0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29856  ts: 0x155A147B573B26B2 OLD: ts: 0x155A147B573B3E22 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2077 data: 0xC0C874A2 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29858  ts: 0x155A147B577B0E12 OLD: ts: 0x155A147B577B2582 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2099 data: 0xC0C07486 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29830  ts: 0x155A147B579B01C2 OLD: ts: 0x155A147B579B0258 dt: 0x0000000000000096 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2161 data: 0xC0C07479 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29817  ts: 0x155A147B57DAF8C2 OLD: ts: 0x155A147B57DB011E dt: 0x000000000000085C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2191 data: 0xC0C07492 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29842  ts: 0x155A147B57FAE4A2 OLD: ts: 0x155A147B57FAE920 dt: 0x000000000000047E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2281 data: 0xC0C0748A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29834  ts: 0x155A147B585AC782 OLD: ts: 0x155A147B585AC962 dt: 0x00000000000001E0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2349 data: 0xC0C874A5 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29861  ts: 0x155A147B589AA712 OLD: ts: 0x155A147B589ABE82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2409 data: 0xC0C874A9 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29865  ts: 0x155A147B58DA8E72 OLD: ts: 0x155A147B58DAA5E2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2489 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B593A6982 OLD: ts: 0x155A147B593A69D2 dt: 0x0000000000000050 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2583 data: 0xC0C07480 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29824  ts: 0x155A147B599A4C62 OLD: ts: 0x155A147B599A4DFC dt: 0x000000000000019A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2641 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B59DA33C2 OLD: ts: 0x155A147B59DA38CC dt: 0x000000000000050A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2673 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B59FA2F42 OLD: ts: 0x155A147B59FA3A96 dt: 0x0000000000000B54 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2733 data: 0xC0C0748B module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29835  ts: 0x155A147B5A3A16A2 OLD: ts: 0x155A147B5A3A18FA dt: 0x0000000000000258 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2801 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B5A79F632 OLD: ts: 0x155A147B5A7A0DA2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2861 data: 0xC0C8749C module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29852  ts: 0x155A147B5AB9DD92 OLD: ts: 0x155A147B5AB9F502 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3011 data: 0xC0C874A0 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29856  ts: 0x155A147B5B59A7D2 OLD: ts: 0x155A147B5B59BF42 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3073 data: 0xC0C874AA module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29866  ts: 0x155A147B5B998F32 OLD: ts: 0x155A147B5B99A6A2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3155 data: 0xC0C07484 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29828  ts: 0x155A147B5BF96A42 OLD: ts: 0x155A147B5BF96C18 dt: 0x00000000000001D6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3189 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B5C1965C2 OLD: ts: 0x155A147B5C196644 dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3289 data: 0xC0C874A6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29862  ts: 0x155A147B5C7940D2 OLD: ts: 0x155A147B5C795842 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3311 data: 0xC0C0748F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29839  ts: 0x155A147B5C993482 OLD: ts: 0x155A147B5C993662 dt: 0x00000000000001E0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3343 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B5CB93002 OLD: ts: 0x155A147B5CB93C28 dt: 0x0000000000000C26 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3373 data: 0xC0C0748C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29836  ts: 0x155A147B5CD91BE2 OLD: ts: 0x155A147B5CD92060 dt: 0x000000000000047E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3429 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B5D190B12 OLD: ts: 0x155A147B5D192282 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3487 data: 0xC0C874B1 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29873  ts: 0x155A147B5D58F272 OLD: ts: 0x155A147B5D5909E2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3547 data: 0xC0C874B2 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29874  ts: 0x155A147B5D98D9D2 OLD: ts: 0x155A147B5D98F142 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3567 data: 0xC0C07483 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29827  ts: 0x155A147B5DB8CD82 OLD: ts: 0x155A147B5DB8CE9A dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3725 data: 0xC0C874AB module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29867  ts: 0x155A147B5E5897C2 OLD: ts: 0x155A147B5E58AF32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3785 data: 0xC0C874AF module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29871  ts: 0x155A147B5E987F22 OLD: ts: 0x155A147B5E989692 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3809 data: 0xC0C07485 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29829  ts: 0x155A147B5EB872D2 OLD: ts: 0x155A147B5EB881D2 dt: 0x0000000000000F00 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3873 data: 0xC0C8749D module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29853  ts: 0x155A147B5EF85A32 OLD: ts: 0x155A147B5EF85F5A dt: 0x0000000000000528 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3903 data: 0xC0C07491 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29841  ts: 0x155A147B5F1855B2 OLD: ts: 0x155A147B5F1859EA dt: 0x0000000000000438 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4001 data: 0xC0C87492 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29842  ts: 0x155A147B5F7830C2 OLD: ts: 0x155A147B5F784832 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4083 data: 0xC0C07489 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29833  ts: 0x155A147B5FD80BD2 OLD: ts: 0x155A147B5FD80C68 dt: 0x0000000000000096 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4211 data: 0xC0C874BA module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29882  ts: 0x155A147B6057E262 OLD: ts: 0x155A147B6057F9D2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4235 data: 0xC0C0747D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29821  ts: 0x155A147B6077D612 OLD: ts: 0x155A147B6077DFA4 dt: 0x0000000000000992 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4323 data: 0xC0C0749C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29852  ts: 0x155A147B60D7B8F2 OLD: ts: 0x155A147B60D7BDDE dt: 0x00000000000004EC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4353 data: 0xC0C07473 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29811  ts: 0x155A147B60F7B472 OLD: ts: 0x155A147B60F7B832 dt: 0x00000000000003C0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4421 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B61379402 OLD: ts: 0x155A147B6137AB72 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4443 data: 0xC0C07486 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29830  ts: 0x155A147B615787B2 OLD: ts: 0x155A147B6157892E dt: 0x000000000000017C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4473 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B61778332 OLD: ts: 0x155A147B61778418 dt: 0x00000000000000E6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4575 data: 0xC0C87496 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29846  ts: 0x155A147B61D75E42 OLD: ts: 0x155A147B61D775B2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4657 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B62373952 OLD: ts: 0x155A147B62373C90 dt: 0x000000000000033E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4687 data: 0xC0C07481 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29825  ts: 0x155A147B625734D2 OLD: ts: 0x155A147B6257384C dt: 0x000000000000037A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4717 data: 0xC0C07497 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29847  ts: 0x155A147B62773052 OLD: ts: 0x155A147B6277346C dt: 0x000000000000041A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4777 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B62B717B2 OLD: ts: 0x155A147B62B71BD6 dt: 0x0000000000000424 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4845 data: 0xC0C87494 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29844  ts: 0x155A147B62F6F742 OLD: ts: 0x155A147B62F70EB2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4905 data: 0xC0C87498 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29848  ts: 0x155A147B6336DEA2 OLD: ts: 0x155A147B6336F612 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5019 data: 0xC0C0749D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29853  ts: 0x155A147B63B6B532 OLD: ts: 0x155A147B63B6B9CE dt: 0x000000000000049C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5057 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B63D6A8E2 OLD: ts: 0x155A147B63D6C052 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5137 data: 0xC0C07494 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29844  ts: 0x155A147B643683F2 OLD: ts: 0x155A147B643688B6 dt: 0x00000000000004C4 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5271 data: 0xC0C874A3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29859  ts: 0x155A147B64B65A82 OLD: ts: 0x155A147B64B671F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5329 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B64F641E2 OLD: ts: 0x155A147B64F65952 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5387 data: 0xC0C87498 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29848  ts: 0x155A147B65362942 OLD: ts: 0x155A147B653640B2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5471 data: 0xC0C0747C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29820  ts: 0x155A147B659613F2 OLD: ts: 0x155A147B65961DD4 dt: 0x00000000000009E2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5531 data: 0xC0C07488 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29832  ts: 0x155A147B65D5FB52 OLD: ts: 0x155A147B65D5FBCA dt: 0x0000000000000078 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  5601 data: 0xC0C87494 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29844  ts: 0x155A147B6615DAE2 OLD: ts: 0x155A147B6615F252 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5661 data: 0xC0C87493 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29843  ts: 0x155A147B6655C242 OLD: ts: 0x155A147B6655D9B2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:    43 data: 0xC0C0748C module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29836  ts: 0x155A147B66F59452 OLD: ts: 0x155A147B66F595C4 dt: 0x0000000000000172 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   115 data: 0xC0C874A8 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29864  ts: 0x155A147B673573E2 OLD: ts: 0x155A147B67358B52 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   175 data: 0xC0C874AA module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29866  ts: 0x155A147B67755B42 OLD: ts: 0x155A147B677572B2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   205 data: 0xC0C87314 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29460  ts: 0x155A147B679556C2 OLD: ts: 0x155A147B67956E32 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   325 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B68152582 OLD: ts: 0x155A147B68153CF2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   437 data: 0xC0C0747A module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29818  ts: 0x155A147B6894FC12 OLD: ts: 0x155A147B6894FC3A dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   597 data: 0xC0C874A9 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29865  ts: 0x155A147B6934BE82 OLD: ts: 0x155A147B6934D5F2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   677 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B6994A932 OLD: ts: 0x155A147B6994AC7A dt: 0x0000000000000348 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   745 data: 0xC0C874A6 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29862  ts: 0x155A147B69D488C2 OLD: ts: 0x155A147B69D4A032 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   801 data: 0xC0C07483 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29827  ts: 0x155A147B6A1477F2 OLD: ts: 0x155A147B6A147AD6 dt: 0x00000000000002E4 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   861 data: 0xC0C0748D module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29837  ts: 0x155A147B6A546722 OLD: ts: 0x155A147B6A546998 dt: 0x0000000000000276 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   929 data: 0xC0C874A3 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29859  ts: 0x155A147B6A9446B2 OLD: ts: 0x155A147B6A945E22 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   989 data: 0xC0C8749F module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29855  ts: 0x155A147B6AD42E12 OLD: ts: 0x155A147B6AD44582 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1049 data: 0xC0C87492 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29842  ts: 0x155A147B6B141572 OLD: ts: 0x155A147B6B142CE2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1071 data: 0xC0C0747F module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29823  ts: 0x155A147B6B340922 OLD: ts: 0x155A147B6B340BCA dt: 0x00000000000002A8 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1129 data: 0xC0C07486 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29830  ts: 0x155A147B6B73F082 OLD: ts: 0x155A147B6B73F370 dt: 0x00000000000002EE tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1191 data: 0xC0C07497 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29847  ts: 0x155A147B6BB3E782 OLD: ts: 0x155A147B6BB3E908 dt: 0x0000000000000186 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1257 data: 0xC0C8749B module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29851  ts: 0x155A147B6BF3C712 OLD: ts: 0x155A147B6BF3DE82 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1279 data: 0xC0C0748E module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29838  ts: 0x155A147B6C13BAC2 OLD: ts: 0x155A147B6C13BB26 dt: 0x0000000000000064 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1407 data: 0xC0C8749A module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29850  ts: 0x155A147B6C939152 OLD: ts: 0x155A147B6C93A8C2 dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    47 ptr:  1641 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B6D93315C OLD: ts: 0x155A147B6D933E9A dt: 0x0000000000000D3E tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1679 data: 0xC0C874AC module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29868  ts: 0x155A147B6DB32A52 OLD: ts: 0x155A147B6DB341C2 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1739 data: 0xC0C874B7 module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29879  ts: 0x155A147B6DF311B2 OLD: ts: 0x155A147B6DF32922 dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:  1761 data: 0xC0C07473 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 29811  ts: 0x155A147B6E130562 OLD: ts: 0x155A147B6E130940 dt: 0x00000000000003DE tag: 2 type: 6
 ***            ADC data: block:    47 ptr:  1937 data: 0xC0C8749C module:  3 fail: 0 range: 0 id:   200 channel:  8 adc: 29852  ts: 0x155A147B6ED2C352 OLD: ts: 0x155A147B6ED2DAC2 dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    47 ptr:  1987 data: 0x83610100 module:  3 information type:  6 information field: 0x00010100 ts: 0x155A147B6F12AC38 OLD: ts: 0x155A147B6F12BA48 dt: 0x0000000000000E10 tag: 2 type: 6
... 43612 more lines ...
Attachment 7: FEE3.out
 *** TDR format 3.3.0 analyser - TD - May 2018
 *** FEE64 discriminator: block:    43 ptr:   709 data: 0x84620001 module:  4 information type:  6 information field: 0x00020001 ts: 0x155A147B1452DAA2 OLD: ts: 0x155A147B1452EBDC dt: 0x000000000000113A tag: 2 type: 6
 ***            ADC data: block:    43 ptr:   771 data: 0xC108752B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29995  ts: 0x155A147B1492CA9A OLD: ts: 0x155A147B1492E20A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:   841 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B14F2A5AA OLD: ts: 0x155A147B14F2A848 dt: 0x000000000000029E tag: 2 type: 6
 ***            ADC data: block:    43 ptr:   953 data: 0xC1087526 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29990  ts: 0x155A147B15727C3A OLD: ts: 0x155A147B157293AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1007 data: 0xC108750B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29963  ts: 0x155A147B15B2639A OLD: ts: 0x155A147B15B27B0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1079 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B16123EAA OLD: ts: 0x155A147B1612430A dt: 0x0000000000000460 tag: 2 type: 6
 *** FEE64 discriminator: block:    43 ptr:  1127 data: 0x84620001 module:  4 information type:  6 information field: 0x00020001 ts: 0x155A147B16522678 OLD: ts: 0x155A147B16523690 dt: 0x0000000000001018 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1191 data: 0xC1087517 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29975  ts: 0x155A147B1692153A OLD: ts: 0x155A147B16922CAA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1247 data: 0xC1087518 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29976  ts: 0x155A147B16D1FC9A OLD: ts: 0x155A147B16D2140A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1265 data: 0xC10074FD module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29949  ts: 0x155A147B16F1F04A OLD: ts: 0x155A147B16F1F234 dt: 0x00000000000001EA tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1291 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B1711EBCA OLD: ts: 0x155A147B1711EC4C dt: 0x0000000000000082 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1375 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B1771C6DA OLD: ts: 0x155A147B1771DE4A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1427 data: 0xC108751D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29981  ts: 0x155A147B17B1AE3A OLD: ts: 0x155A147B17B1C5AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1479 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B17F1959A OLD: ts: 0x155A147B17F1AD0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1497 data: 0xC1007508 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29960  ts: 0x155A147B1811894A OLD: ts: 0x155A147B18118990 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1525 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B183184CA OLD: ts: 0x155A147B183184D4 dt: 0x000000000000000A tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1611 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B18915FDA OLD: ts: 0x155A147B1891774A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1707 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B1911366A OLD: ts: 0x155A147B19113B1A dt: 0x00000000000004B0 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1851 data: 0xC1087517 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29975  ts: 0x155A147B19B0F8DA OLD: ts: 0x155A147B19B1104A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1871 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B19D0EC8A OLD: ts: 0x155A147B19D0FC2A dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  1921 data: 0xC10074ED module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29933  ts: 0x155A147B1A10D3EA OLD: ts: 0x155A147B1A10D4E4 dt: 0x00000000000000FA tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  1999 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B1A70B6CA OLD: ts: 0x155A147B1A70B7BA dt: 0x00000000000000F0 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2023 data: 0xC1007501 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29953  ts: 0x155A147B1A90B24A OLD: ts: 0x155A147B1A90B3C6 dt: 0x000000000000017C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2083 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B1AD091DA OLD: ts: 0x155A147B1AD0A94A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2137 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B1B10793A OLD: ts: 0x155A147B1B1090AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2155 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B1B306CEA OLD: ts: 0x155A147B1B306D30 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2291 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B1BD0372A OLD: ts: 0x155A147B1BD04E9A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2479 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B1CAFE8CA OLD: ts: 0x155A147B1CB0003A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2525 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B1CEFD7FA OLD: ts: 0x155A147B1CEFE79A dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2549 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B1D0FC3DA OLD: ts: 0x155A147B1D0FC6F0 dt: 0x0000000000000316 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2603 data: 0xC1007504 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29956  ts: 0x155A147B1D4FAB3A OLD: ts: 0x155A147B1D4FAB58 dt: 0x000000000000001E tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2629 data: 0xC10074EA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29930  ts: 0x155A147B1D6FA6BA OLD: ts: 0x155A147B1D6FA8EA dt: 0x0000000000000230 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2653 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B1D8FA23A OLD: ts: 0x155A147B1D8FA5D2 dt: 0x0000000000000398 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2713 data: 0xC1087520 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29984  ts: 0x155A147B1DCF81CA OLD: ts: 0x155A147B1DCF993A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2731 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B1DEF757A OLD: ts: 0x155A147B1DEF782C dt: 0x00000000000002B2 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2759 data: 0xC10074EA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29930  ts: 0x155A147B1E0F70FA OLD: ts: 0x155A147B1E0F7C26 dt: 0x0000000000000B2C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2897 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B1EAF336A OLD: ts: 0x155A147B1EAF4ADA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  2969 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B1F0F0E7A OLD: ts: 0x155A147B1F0F10E6 dt: 0x000000000000026C tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  2995 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B1F2F09FA OLD: ts: 0x155A147B1F2F0B3A dt: 0x0000000000000140 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3073 data: 0xC1087519 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29977  ts: 0x155A147B1F8EE50A OLD: ts: 0x155A147B1F8EFC7A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3125 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B1FCECC6A OLD: ts: 0x155A147B1FCEE3DA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3221 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B204EA2FA OLD: ts: 0x155A147B204EA822 dt: 0x0000000000000528 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3309 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B20AE7E0A OLD: ts: 0x155A147B20AE957A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3361 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B20EE656A OLD: ts: 0x155A147B20EE7CDA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3377 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B210E591A OLD: ts: 0x155A147B210E5C3A dt: 0x0000000000000320 tag: 2 type: 6
 ***            ADC data: block:    43 ptr:  3463 data: 0xC1387437 module:  4 fail: 0 range: 0 id:   312 channel: 56 adc: 29751  ts: 0x155A147B216E3BFA OLD: ts: 0x155A147B216E536A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3489 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B218E2FAA OLD: ts: 0x155A147B218E471A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    43 ptr:  3507 data: 0xC1007500 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29952  ts: 0x155A147B21AE235A OLD: ts: 0x155A147B21AE23BE dt: 0x0000000000000064 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:    19 data: 0xC10074F0 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29936  ts: 0x155A147B21CE1EDA OLD: ts: 0x155A147B21CE2628 dt: 0x000000000000074E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:    71 data: 0xC10074FD module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29949  ts: 0x155A147B220E063A OLD: ts: 0x155A147B220E095A dt: 0x0000000000000320 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:    97 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B222E01BA OLD: ts: 0x155A147B222E03EA dt: 0x0000000000000230 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   209 data: 0xC1087516 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29974  ts: 0x155A147B22ADC8AA OLD: ts: 0x155A147B22ADE01A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   261 data: 0xC1087525 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29989  ts: 0x155A147B22EDB00A OLD: ts: 0x155A147B22EDC77A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   279 data: 0xC10074F6 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29942  ts: 0x155A147B230DA3BA OLD: ts: 0x155A147B230DB17A dt: 0x0000000000000DC0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   393 data: 0xC1087516 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29974  ts: 0x155A147B238D7A4A OLD: ts: 0x155A147B238D91BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   565 data: 0xC10074FD module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29949  ts: 0x155A147B246D33BA OLD: ts: 0x155A147B246D35E0 dt: 0x0000000000000226 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:   631 data: 0xC1087517 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29975  ts: 0x155A147B24AD134A OLD: ts: 0x155A147B24AD2ABA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   763 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B254CDD8A OLD: ts: 0x155A147B254CF4FA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   815 data: 0xC1087524 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29988  ts: 0x155A147B258CC4EA OLD: ts: 0x155A147B258CDC5A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:   989 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B266C768A OLD: ts: 0x155A147B266C8DFA dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    44 ptr:  1109 data: 0x84630001 module:  4 information type:  6 information field: 0x00030001 ts: 0x155A147B270C3602 OLD: ts: 0x155A147B270C48E0 dt: 0x00000000000012DE tag: 2 type: 6
 *** FEE64 discriminator: block:    44 ptr:  1137 data: 0x84610100 module:  4 information type:  6 information field: 0x00010100 ts: 0x155A147B272C2BB0 OLD: ts: 0x155A147B272C3F9C dt: 0x00000000000013EC tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1177 data: 0xC1087515 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29973  ts: 0x155A147B274C282A OLD: ts: 0x155A147B274C3F9A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1203 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B276C1BDA OLD: ts: 0x155A147B276C334A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1221 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B278C0F8A OLD: ts: 0x155A147B278C102A dt: 0x00000000000000A0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1335 data: 0xC1087511 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29969  ts: 0x155A147B280BE61A OLD: ts: 0x155A147B280BFD8A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1387 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B284BCD7A OLD: ts: 0x155A147B284BE4EA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1433 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B288BBCAA OLD: ts: 0x155A147B288BBDC2 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1487 data: 0xC10074EE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29934  ts: 0x155A147B28CBA40A OLD: ts: 0x155A147B28CBA4A0 dt: 0x0000000000000096 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1515 data: 0xC1007509 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29961  ts: 0x155A147B28EB9F8A OLD: ts: 0x155A147B28EBA14C dt: 0x00000000000001C2 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1575 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B292B7F1A OLD: ts: 0x155A147B292B968A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1623 data: 0xC11073F2 module:  4 fail: 0 range: 0 id:   272 channel: 16 adc: 29682  ts: 0x155A147B296B6E4A OLD: ts: 0x155A147B296B7DEA dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1721 data: 0xC10074F0 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29936  ts: 0x155A147B29EB3D0A OLD: ts: 0x155A147B29EB3ECC dt: 0x00000000000001C2 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1753 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B2A0B388A OLD: ts: 0x155A147B2A0B3BDC dt: 0x0000000000000352 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1813 data: 0xC1087525 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29989  ts: 0x155A147B2A4B181A OLD: ts: 0x155A147B2A4B2F8A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  1883 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B2AAAF32A OLD: ts: 0x155A147B2AAAF7D0 dt: 0x00000000000004A6 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1933 data: 0xC12073B9 module:  4 fail: 0 range: 0 id:   288 channel: 32 adc: 29625  ts: 0x155A147B2AEADA8A OLD: ts: 0x155A147B2AEADFC6 dt: 0x000000000000053C tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  1959 data: 0xC1007500 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29952  ts: 0x155A147B2B0AD60A OLD: ts: 0x155A147B2B0AD650 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2047 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B2B6AB11A OLD: ts: 0x155A147B2B6AC88A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2101 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B2BAA987A OLD: ts: 0x155A147B2BAAAFEA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2119 data: 0xC10074F0 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29936  ts: 0x155A147B2BCA8C2A OLD: ts: 0x155A147B2BCA8FAE dt: 0x0000000000000384 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2179 data: 0xC1087524 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29988  ts: 0x155A147B2C0A7B5A OLD: ts: 0x155A147B2C0A92CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2309 data: 0xC1087527 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29991  ts: 0x155A147B2CAA459A OLD: ts: 0x155A147B2CAA5D0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2365 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B2CEA2CFA OLD: ts: 0x155A147B2CEA446A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2511 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B2DA9EAEA OLD: ts: 0x155A147B2DA9EFEA dt: 0x0000000000000500 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2543 data: 0xC1087518 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29976  ts: 0x155A147B2DC9DE9A OLD: ts: 0x155A147B2DC9F60A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2595 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B2E09C5FA OLD: ts: 0x155A147B2E09DD6A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2663 data: 0xC10074F5 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29941  ts: 0x155A147B2E69A10A OLD: ts: 0x155A147B2E69A5B0 dt: 0x00000000000004A6 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  2777 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B2EE9779A OLD: ts: 0x155A147B2EE98F0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  2875 data: 0xC1007500 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29952  ts: 0x155A147B2F694E2A OLD: ts: 0x155A147B2F695316 dt: 0x00000000000004EC tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3017 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B3009109A OLD: ts: 0x155A147B3009280A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3119 data: 0xC1087515 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29973  ts: 0x155A147B3088DF5A OLD: ts: 0x155A147B3088F6CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3137 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B30A8D30A OLD: ts: 0x155A147B30A8D490 dt: 0x0000000000000186 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3239 data: 0xC10074F6 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29942  ts: 0x155A147B3128B16A OLD: ts: 0x155A147B3128B48A dt: 0x0000000000000320 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3301 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B316890FA OLD: ts: 0x155A147B3168A86A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3369 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B31C86C0A OLD: ts: 0x155A147B31C87132 dt: 0x0000000000000528 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3451 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B32284EEA OLD: ts: 0x155A147B32285D68 dt: 0x0000000000000E7E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3477 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B32484A6A OLD: ts: 0x155A147B32484C2C dt: 0x00000000000001C2 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3533 data: 0xC10074ED module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29933  ts: 0x155A147B328831CA OLD: ts: 0x155A147B328834E0 dt: 0x0000000000000316 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3593 data: 0xC1087530 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 30000  ts: 0x155A147B32C8115A OLD: ts: 0x155A147B32C828CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3687 data: 0xC1087526 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29990  ts: 0x155A147B3347E7EA OLD: ts: 0x155A147B3347ED1C dt: 0x0000000000000532 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3799 data: 0xC1087515 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29973  ts: 0x155A147B33C7B6AA OLD: ts: 0x155A147B33C7CE1A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  3893 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B34478D3A OLD: ts: 0x155A147B34478FD8 dt: 0x000000000000029E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3919 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B346788BA OLD: ts: 0x155A147B346788D8 dt: 0x000000000000001E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  3979 data: 0xC108751F module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29983  ts: 0x155A147B34A7684A OLD: ts: 0x155A147B34A77FBA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4047 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B3507435A OLD: ts: 0x155A147B3507463E dt: 0x00000000000002E4 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4077 data: 0xC10074E7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29927  ts: 0x155A147B35273EDA OLD: ts: 0x155A147B35273FC0 dt: 0x00000000000000E6 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4159 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B358721BA OLD: ts: 0x155A147B358722AA dt: 0x00000000000000F0 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4217 data: 0xC108751B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29979  ts: 0x155A147B35C7014A OLD: ts: 0x155A147B35C718BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4271 data: 0xC1087525 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29989  ts: 0x155A147B3606E8AA OLD: ts: 0x155A147B3607001A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4291 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B3626DC5A OLD: ts: 0x155A147B3626E8B2 dt: 0x0000000000000C58 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4317 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B3646D7DA OLD: ts: 0x155A147B3646D8E8 dt: 0x000000000000010E tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4393 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B36A6BABA OLD: ts: 0x155A147B36A6BD30 dt: 0x0000000000000276 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4453 data: 0xC1087525 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29989  ts: 0x155A147B36E69A4A OLD: ts: 0x155A147B36E6B1BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4471 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B37068DFA OLD: ts: 0x155A147B37068EA4 dt: 0x00000000000000AA tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4575 data: 0xC1087521 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29985  ts: 0x155A147B3786648A OLD: ts: 0x155A147B37867BFA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4625 data: 0xC1387437 module:  4 fail: 0 range: 0 id:   312 channel: 56 adc: 29751  ts: 0x155A147B37C653BA OLD: ts: 0x155A147B37C6635A dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4725 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B3846227A OLD: ts: 0x155A147B3846269E dt: 0x0000000000000424 tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  4811 data: 0xC108751B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29979  ts: 0x155A147B38A5FD8A OLD: ts: 0x155A147B38A614FA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4863 data: 0xC1087519 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29977  ts: 0x155A147B38E5E4EA OLD: ts: 0x155A147B38E5FC5A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    44 ptr:  4881 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B3905D89A OLD: ts: 0x155A147B3905DC78 dt: 0x00000000000003DE tag: 2 type: 6
 ***            ADC data: block:    44 ptr:  5051 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B39C5968A OLD: ts: 0x155A147B39C5ADFA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:    49 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B3A057DEA OLD: ts: 0x155A147B3A05955A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:    67 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B3A25719A OLD: ts: 0x155A147B3A257596 dt: 0x00000000000003FC tag: 2 type: 6
 ***            ADC data: block:    45 ptr:    95 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B3A456D1A OLD: ts: 0x155A147B3A456EAA dt: 0x0000000000000190 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   123 data: 0xC100750B module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29963  ts: 0x155A147B3A65689A OLD: ts: 0x155A147B3A6569D0 dt: 0x0000000000000136 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   189 data: 0xC1087521 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29985  ts: 0x155A147B3AA5482A OLD: ts: 0x155A147B3AA55F9A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   205 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B3AC53BDA OLD: ts: 0x155A147B3AC53CFC dt: 0x0000000000000122 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   317 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B3B45126A OLD: ts: 0x155A147B3B4529DA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   385 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B3BA4ED7A OLD: ts: 0x155A147B3BA4EF32 dt: 0x00000000000001B8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   467 data: 0xC1007503 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29955  ts: 0x155A147B3C04D05A OLD: ts: 0x155A147B3C04D4F6 dt: 0x000000000000049C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   493 data: 0xC10074F5 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29941  ts: 0x155A147B3C24CBDA OLD: ts: 0x155A147B3C24CF54 dt: 0x000000000000037A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   553 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B3C64AB6A OLD: ts: 0x155A147B3C64C2DA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   683 data: 0xC1087502 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29954  ts: 0x155A147B3D0475AA OLD: ts: 0x155A147B3D048D1A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   735 data: 0xC1087523 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29987  ts: 0x155A147B3D445D0A OLD: ts: 0x155A147B3D44747A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:   785 data: 0xC1007501 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29953  ts: 0x155A147B3D844C3A OLD: ts: 0x155A147B3D844D3E dt: 0x0000000000000104 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   929 data: 0xC1007504 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29956  ts: 0x155A147B3E44025A OLD: ts: 0x155A147B3E440638 dt: 0x00000000000003DE tag: 2 type: 6
 ***            ADC data: block:    45 ptr:   955 data: 0xC10074EE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29934  ts: 0x155A147B3E63FDDA OLD: ts: 0x155A147B3E640140 dt: 0x0000000000000366 tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:   979 data: 0x84620001 module:  4 information type:  6 information field: 0x00020001 ts: 0x155A147B3E83E8F2 OLD: ts: 0x155A147B3E83F8B0 dt: 0x0000000000000FBE tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1043 data: 0xC1087515 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29973  ts: 0x155A147B3EC3D8EA OLD: ts: 0x155A147B3EC3F05A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1095 data: 0xC1087519 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29977  ts: 0x155A147B3F03C04A OLD: ts: 0x155A147B3F03D7BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1115 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B3F23B3FA OLD: ts: 0x155A147B3F23BBA2 dt: 0x00000000000007A8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1141 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B3F43A7AA OLD: ts: 0x155A147B3F43A98A dt: 0x00000000000001E0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1177 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B3F639B5A OLD: ts: 0x155A147B3F63B2CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1195 data: 0xC1007506 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29958  ts: 0x155A147B3F838F0A OLD: ts: 0x155A147B3F839086 dt: 0x000000000000017C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1221 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B3FA38A8A OLD: ts: 0x155A147B3FA39DD6 dt: 0x000000000000134C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1247 data: 0xC1007504 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29956  ts: 0x155A147B3FC3860A OLD: ts: 0x155A147B3FC388BC dt: 0x00000000000002B2 tag: 2 type: 6
 *** FEE64 discriminator: block:    45 ptr:  1291 data: 0x84610100 module:  4 information type:  6 information field: 0x00010100 ts: 0x155A147B30035B22 OLD: ts: 0x155A147B3FF45604 dt: 0x000000000FF0FAE2 tag: 2 type: 4
 ***            ADC data: block:    45 ptr:  1309 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B3003659A OLD: ts: 0x155A147B30037D0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1409 data: 0xC100750A module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29962  ts: 0x155A147B40833C2A OLD: ts: 0x155A147B40834044 dt: 0x000000000000041A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1435 data: 0xC10074E9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29929  ts: 0x155A147B40A337AA OLD: ts: 0x155A147B40A3387C dt: 0x00000000000000D2 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1495 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B40E3173A OLD: ts: 0x155A147B40E32EAA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1591 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B4162EDCA OLD: ts: 0x155A147B4162F13A dt: 0x0000000000000370 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1617 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B4182E94A OLD: ts: 0x155A147B4182EC06 dt: 0x00000000000002BC tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  1681 data: 0xC1087511 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29969  ts: 0x155A147B41C2C8DA OLD: ts: 0x155A147B41C2E04A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1707 data: 0xC138743F module:  4 fail: 0 range: 0 id:   312 channel: 56 adc: 29759  ts: 0x155A147B41E2BC8A OLD: ts: 0x155A147B41E2D3FA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1733 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B4202B03A OLD: ts: 0x155A147B4202C7AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1787 data: 0xC1087516 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29974  ts: 0x155A147B4242979A OLD: ts: 0x155A147B4242AF0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  1917 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B42E261DA OLD: ts: 0x155A147B42E2794A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2047 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B43822C1A OLD: ts: 0x155A147B4382438A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2167 data: 0xC10074FB module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29947  ts: 0x155A147B4421FE2A OLD: ts: 0x155A147B4422008C dt: 0x0000000000000262 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2231 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B4461DDBA OLD: ts: 0x155A147B4461F52A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2285 data: 0xC108750D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29965  ts: 0x155A147B44A1C51A OLD: ts: 0x155A147B44A1DC8A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2303 data: 0xC10074FC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29948  ts: 0x155A147B44C1B8CA OLD: ts: 0x155A147B44C1B9E2 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2331 data: 0xC10074ED module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29933  ts: 0x155A147B44E1B44A OLD: ts: 0x155A147B44E1C2AA dt: 0x0000000000000E60 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2365 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B4501A7FA OLD: ts: 0x155A147B4501BF6A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2423 data: 0xC1087521 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29985  ts: 0x155A147B45418F5A OLD: ts: 0x155A147B4541A6CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2611 data: 0xC1087523 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29987  ts: 0x155A147B462140FA OLD: ts: 0x155A147B4621586A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2665 data: 0xC1087525 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29989  ts: 0x155A147B4661285A OLD: ts: 0x155A147B46613FCA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2761 data: 0xC10074FB module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29947  ts: 0x155A147B46E0FEEA OLD: ts: 0x155A147B46E10322 dt: 0x0000000000000438 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  2845 data: 0xC1087520 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29984  ts: 0x155A147B4740D9FA OLD: ts: 0x155A147B4740F16A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  2945 data: 0xC10074E9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29929  ts: 0x155A147B47C0B08A OLD: ts: 0x155A147B47C0B7C4 dt: 0x000000000000073A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3031 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B48208B9A OLD: ts: 0x155A147B4820A30A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3085 data: 0xC1087521 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29985  ts: 0x155A147B486072FA OLD: ts: 0x155A147B48608A6A dt: 0x0000000000001770 tag: 3 type: 0
 *** FEE64 discriminator: block:    45 ptr:  3187 data: 0x84610100 module:  4 information type:  6 information field: 0x00010100 ts: 0x155A147B48E03F4E OLD: ts: 0x155A147B48E051D2 dt: 0x0000000000001284 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3223 data: 0xC1087519 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29977  ts: 0x155A147B49003D3A OLD: ts: 0x155A147B490054AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3295 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B4960184A OLD: ts: 0x155A147B49601C96 dt: 0x000000000000044C tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3399 data: 0xC10074F0 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29936  ts: 0x155A147B49DFF6AA OLD: ts: 0x155A147B49DFF858 dt: 0x00000000000001AE tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3459 data: 0xC108750B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29963  ts: 0x155A147B4A1FD63A OLD: ts: 0x155A147B4A1FEDAA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3505 data: 0xC10074EA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29930  ts: 0x155A147B4A5FC56A OLD: ts: 0x155A147B4A5FC934 dt: 0x00000000000003CA tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3531 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B4A7FB14A OLD: ts: 0x155A147B4A7FB564 dt: 0x000000000000041A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3587 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B4ABFA84A OLD: ts: 0x155A147B4ABFB2AE dt: 0x0000000000000A64 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3645 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B4AFF87DA OLD: ts: 0x155A147B4AFF9F4A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3663 data: 0xC1007509 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29961  ts: 0x155A147B4B1F7B8A OLD: ts: 0x155A147B4B1F8B2A dt: 0x0000000000000FA0 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3769 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B4B9F521A OLD: ts: 0x155A147B4B9F558A dt: 0x0000000000000370 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3795 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B4BBF4D9A OLD: ts: 0x155A147B4BBF5092 dt: 0x00000000000002F8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  3855 data: 0xC108751D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29981  ts: 0x155A147B4BFF2D2A OLD: ts: 0x155A147B4BFF449A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3907 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B4C3F148A OLD: ts: 0x155A147B4C3F2BFA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  3957 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B4C7EFBEA OLD: ts: 0x155A147B4C7F135A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4061 data: 0xC108752A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29994  ts: 0x155A147B4CFECAAA OLD: ts: 0x155A147B4CFEE21A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4079 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B4D1EBE5A OLD: ts: 0x155A147B4D1EC012 dt: 0x00000000000001B8 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4159 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B4D7EA13A OLD: ts: 0x155A147B4D7EAF7C dt: 0x0000000000000E42 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4297 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B4E1E63AA OLD: ts: 0x155A147B4E1E7B1A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4317 data: 0xC1007506 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29958  ts: 0x155A147B4E3E575A OLD: ts: 0x155A147B4E3E60C4 dt: 0x000000000000096A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4395 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B4E9E3A3A OLD: ts: 0x155A147B4E9E3C92 dt: 0x0000000000000258 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4421 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B4EBE35BA OLD: ts: 0x155A147B4EBE3614 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4481 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B4EFE154A OLD: ts: 0x155A147B4EFE2CBA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4527 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B4F3E047A OLD: ts: 0x155A147B4F3E056A dt: 0x00000000000000F0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4553 data: 0xC10074FB module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29947  ts: 0x155A147B4F5DF05A OLD: ts: 0x155A147B4F5DF316 dt: 0x00000000000002BC tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4685 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B4FFDBA9A OLD: ts: 0x155A147B4FFDBB94 dt: 0x00000000000000FA tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4773 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B505D95AA OLD: ts: 0x155A147B505DAD1A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  4845 data: 0xC10074EF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29935  ts: 0x155A147B50BD70BA OLD: ts: 0x155A147B50BD756A dt: 0x00000000000004B0 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4871 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B50DD6C3A OLD: ts: 0x155A147B50DD70CC dt: 0x0000000000000492 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4899 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B50FD67BA OLD: ts: 0x155A147B50FD6800 dt: 0x0000000000000046 tag: 2 type: 6
 ***            ADC data: block:    45 ptr:  4961 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B513D474A OLD: ts: 0x155A147B513D5EBA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    45 ptr:  5015 data: 0xC108751B module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29979  ts: 0x155A147B517D2EAA OLD: ts: 0x155A147B517D461A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:    31 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B51DD09BA OLD: ts: 0x155A147B51DD0DC0 dt: 0x0000000000000406 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:    59 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B51FD053A OLD: ts: 0x155A147B51FD0684 dt: 0x000000000000014A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:    87 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B521D00BA OLD: ts: 0x155A147B521D0A38 dt: 0x000000000000097E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   201 data: 0xC108751D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29981  ts: 0x155A147B529CC7AA OLD: ts: 0x155A147B529CDF1A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   219 data: 0xC10074F3 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29939  ts: 0x155A147B52BCBB5A OLD: ts: 0x155A147B52BCBBC8 dt: 0x000000000000006E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   245 data: 0xC10074FB module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29947  ts: 0x155A147B52DCB6DA OLD: ts: 0x155A147B52DCB84C dt: 0x0000000000000172 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   331 data: 0xC108751A module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29978  ts: 0x155A147B533C91EA OLD: ts: 0x155A147B533CA95A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   407 data: 0xC1007505 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29957  ts: 0x155A147B539C6CFA OLD: ts: 0x155A147B539C713C dt: 0x0000000000000442 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   563 data: 0xC1007504 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29956  ts: 0x155A147B545C32BA OLD: ts: 0x155A147B545C354E dt: 0x0000000000000294 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   623 data: 0xC1087528 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29992  ts: 0x155A147B549C124A OLD: ts: 0x155A147B549C29BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   693 data: 0xC10074F0 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29936  ts: 0x155A147B54FBED5A OLD: ts: 0x155A147B54FBF1C4 dt: 0x000000000000046A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   805 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B557BC3EA OLD: ts: 0x155A147B557BDB5A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:   865 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B55DB9EFA OLD: ts: 0x155A147B55DBA1D4 dt: 0x00000000000002DA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   921 data: 0xC1007503 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29955  ts: 0x155A147B561B95FA OLD: ts: 0x155A147B561B9834 dt: 0x000000000000023A tag: 2 type: 6
 ***            ADC data: block:    46 ptr:   985 data: 0xC108751D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29981  ts: 0x155A147B565B758A OLD: ts: 0x155A147B565B8CFA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1037 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B569B5CEA OLD: ts: 0x155A147B569B745A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1053 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B56BB509A OLD: ts: 0x155A147B56BB552C dt: 0x0000000000000492 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1139 data: 0xC1087527 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29991  ts: 0x155A147B571B337A OLD: ts: 0x155A147B571B4AEA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1191 data: 0xC108750C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29964  ts: 0x155A147B575B1ADA OLD: ts: 0x155A147B575B324A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1261 data: 0xC1007506 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29958  ts: 0x155A147B57BAF5EA OLD: ts: 0x155A147B57BAF900 dt: 0x0000000000000316 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1339 data: 0xC10074EF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29935  ts: 0x155A147B581AD8CA OLD: ts: 0x155A147B581ADC58 dt: 0x000000000000038E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1425 data: 0xC1087520 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29984  ts: 0x155A147B587AB3DA OLD: ts: 0x155A147B587ACB4A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1479 data: 0xC1087509 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29961  ts: 0x155A147B58BA9B3A OLD: ts: 0x155A147B58BAB2AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1537 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B58FA829A OLD: ts: 0x155A147B58FA9A0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1613 data: 0xC10074F8 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29944  ts: 0x155A147B595A6D4A OLD: ts: 0x155A147B595A6F98 dt: 0x000000000000024E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1641 data: 0xC10074FE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29950  ts: 0x155A147B597A592A OLD: ts: 0x155A147B597A5C72 dt: 0x0000000000000348 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1695 data: 0xC1007507 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29959  ts: 0x155A147B59BA408A OLD: ts: 0x155A147B59BA4C88 dt: 0x0000000000000BFE tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1745 data: 0xC10074EF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29935  ts: 0x155A147B59FA27EA OLD: ts: 0x155A147B59FA2BB4 dt: 0x00000000000003CA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1771 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B5A1A236A OLD: ts: 0x155A147B5A1A241E dt: 0x00000000000000B4 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  1831 data: 0xC1087512 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29970  ts: 0x155A147B5A5A02FA OLD: ts: 0x155A147B5A5A1A6A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1883 data: 0xC1087529 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29993  ts: 0x155A147B5A99EA5A OLD: ts: 0x155A147B5A9A01CA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  1983 data: 0xC10074FF module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29951  ts: 0x155A147B5B19C0EA OLD: ts: 0x155A147B5B19C5E0 dt: 0x00000000000004F6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2009 data: 0xC10074F5 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29941  ts: 0x155A147B5B39BC6A OLD: ts: 0x155A147B5B39BD82 dt: 0x0000000000000118 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2069 data: 0xC108751E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29982  ts: 0x155A147B5B799BFA OLD: ts: 0x155A147B5B79B36A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2121 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B5BB9835A OLD: ts: 0x155A147B5BB99ACA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2173 data: 0xC10074FC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29948  ts: 0x155A147B5BF9728A OLD: ts: 0x155A147B5BF97B4A dt: 0x00000000000008C0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2227 data: 0xC1007503 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29955  ts: 0x155A147B5C3959EA OLD: ts: 0x155A147B5C395E22 dt: 0x0000000000000438 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2261 data: 0xC1087518 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29976  ts: 0x155A147B5C594D9A OLD: ts: 0x155A147B5C59650A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2281 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B5C79414A OLD: ts: 0x155A147B5C79430C dt: 0x00000000000001C2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2333 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B5CB928AA OLD: ts: 0x155A147B5CB92A76 dt: 0x00000000000001CC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2449 data: 0xC1087522 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29986  ts: 0x155A147B5D38FF3A OLD: ts: 0x155A147B5D3916AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2503 data: 0xC1087515 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29973  ts: 0x155A147B5D78E69A OLD: ts: 0x155A147B5D78FE0A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2575 data: 0xC1007501 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29953  ts: 0x155A147B5DD8C1AA OLD: ts: 0x155A147B5DD8C4F2 dt: 0x0000000000000348 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2603 data: 0xC10074ED module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29933  ts: 0x155A147B5DF8BD2A OLD: ts: 0x155A147B5DF8BDF2 dt: 0x00000000000000C8 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2629 data: 0xC10074FD module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29949  ts: 0x155A147B5E18B8AA OLD: ts: 0x155A147B5E18B8D2 dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2691 data: 0xC1087523 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29987  ts: 0x155A147B5E58983A OLD: ts: 0x155A147B5E58AFAA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2749 data: 0xC108751D module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29981  ts: 0x155A147B5E987F9A OLD: ts: 0x155A147B5E98970A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  2769 data: 0xC10074F1 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29937  ts: 0x155A147B5EB8734A OLD: ts: 0x155A147B5EB87476 dt: 0x000000000000012C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2823 data: 0xC10074EE module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29934  ts: 0x155A147B5EF85AAA OLD: ts: 0x155A147B5EF85F50 dt: 0x00000000000004A6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2849 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B5F18562A OLD: ts: 0x155A147B5F1859EA dt: 0x00000000000003C0 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  2937 data: 0xC1087510 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29968  ts: 0x155A147B5F78313A OLD: ts: 0x155A147B5F7848AA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3009 data: 0xC1007505 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29957  ts: 0x155A147B5FD80C4A OLD: ts: 0x155A147B5FD80C5E dt: 0x0000000000000014 tag: 2 type: 6
 *** FEE64 discriminator: block:    46 ptr:  3087 data: 0x84610100 module:  4 information type:  6 information field: 0x00010100 ts: 0x155A147B6037E638 OLD: ts: 0x155A147B6037FA74 dt: 0x000000000000143C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3123 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B6057E2DA OLD: ts: 0x155A147B6057FA4A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3195 data: 0xC10074ED module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29933  ts: 0x155A147B60B7CD8A OLD: ts: 0x155A147B60B7D2EE dt: 0x0000000000000564 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3219 data: 0xC1007503 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29955  ts: 0x155A147B60D7B96A OLD: ts: 0x155A147B60D7BDDE dt: 0x0000000000000474 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3247 data: 0xC10074EC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29932  ts: 0x155A147B60F7B4EA OLD: ts: 0x155A147B60F7B828 dt: 0x000000000000033E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3309 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B6137947A OLD: ts: 0x155A147B6137ABEA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3333 data: 0xC10074F6 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29942  ts: 0x155A147B6157882A OLD: ts: 0x155A147B61578924 dt: 0x00000000000000FA tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3359 data: 0xC10074F4 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29940  ts: 0x155A147B617783AA OLD: ts: 0x155A147B61778418 dt: 0x000000000000006E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3443 data: 0xC1087513 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29971  ts: 0x155A147B61D75EBA OLD: ts: 0x155A147B61D7762A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3513 data: 0xC1007504 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29956  ts: 0x155A147B623739CA OLD: ts: 0x155A147B62373C90 dt: 0x00000000000002C6 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3539 data: 0xC10074F5 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29941  ts: 0x155A147B6257354A OLD: ts: 0x155A147B6257384C dt: 0x0000000000000302 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3567 data: 0xC1007507 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29959  ts: 0x155A147B627730CA OLD: ts: 0x155A147B62773462 dt: 0x0000000000000398 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3619 data: 0xC10074FC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29948  ts: 0x155A147B62B7182A OLD: ts: 0x155A147B62B71BD6 dt: 0x00000000000003AC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3681 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B62F6F7BA OLD: ts: 0x155A147B62F70F2A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3707 data: 0xC108752C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29996  ts: 0x155A147B6316EB6A OLD: ts: 0x155A147B631702DA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  3753 data: 0xC10074FB module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29947  ts: 0x155A147B6356DA9A OLD: ts: 0x155A147B6356DF36 dt: 0x000000000000049C tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3805 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B6396C1FA OLD: ts: 0x155A147B6396C222 dt: 0x0000000000000028 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  3891 data: 0xC1087523 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29987  ts: 0x155A147B63F69D0A OLD: ts: 0x155A147B63F6B47A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4025 data: 0xC108750E module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29966  ts: 0x155A147B6496674A OLD: ts: 0x155A147B64967EBA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4077 data: 0xC1087520 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29984  ts: 0x155A147B64D64EAA OLD: ts: 0x155A147B64D6661A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4131 data: 0xC1087524 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29988  ts: 0x155A147B6516360A OLD: ts: 0x155A147B65164D7A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4183 data: 0xC1087514 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29972  ts: 0x155A147B65561D6A OLD: ts: 0x155A147B655634DA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4201 data: 0xC10074FA module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29946  ts: 0x155A147B6576111A OLD: ts: 0x155A147B65761318 dt: 0x00000000000001FE tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4305 data: 0xC10074FC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29948  ts: 0x155A147B65F5EF7A OLD: ts: 0x155A147B65F5F218 dt: 0x000000000000029E tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4365 data: 0xC108751F module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29983  ts: 0x155A147B6635CF0A OLD: ts: 0x155A147B6635E67A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4437 data: 0xC1007505 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29957  ts: 0x155A147B6695AA1A OLD: ts: 0x155A147B6695AC72 dt: 0x0000000000000258 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4463 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B66B5A59A OLD: ts: 0x155A147B66B5AA54 dt: 0x00000000000004BA tag: 2 type: 6
 *** FEE64 discriminator: block:    46 ptr:  4493 data: 0x84620001 module:  4 information type:  6 information field: 0x00020001 ts: 0x155A147B66D58EF0 OLD: ts: 0x155A147B66D59F26 dt: 0x0000000000001036 tag: 2 type: 6
 *** FEE64 discriminator: block:    46 ptr:  4523 data: 0x84630001 module:  4 information type:  6 information field: 0x00030001 ts: 0x155A147B66F58232 OLD: ts: 0x155A147B66F595BA dt: 0x0000000000001388 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4601 data: 0xC108750C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29964  ts: 0x155A147B6755680A OLD: ts: 0x155A147B67557F7A dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  4669 data: 0xC10074FD module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29949  ts: 0x155A147B67B5431A OLD: ts: 0x155A147B67B546BC dt: 0x00000000000003A2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4695 data: 0xC10074F2 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29938  ts: 0x155A147B67D53E9A OLD: ts: 0x155A147B67D53F6C dt: 0x00000000000000D2 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4793 data: 0xC1007500 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29952  ts: 0x155A147B68550D5A OLD: ts: 0x155A147B68550E90 dt: 0x0000000000000136 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4819 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B687508DA OLD: ts: 0x155A147B68750AA6 dt: 0x00000000000001CC tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4871 data: 0xC1007506 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29958  ts: 0x155A147B68B4F03A OLD: ts: 0x155A147B68B50142 dt: 0x0000000000001108 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4897 data: 0xC10074EC module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29932  ts: 0x155A147B68D4EBBA OLD: ts: 0x155A147B68D4ED40 dt: 0x0000000000000186 tag: 2 type: 6
 ***            ADC data: block:    46 ptr:  4957 data: 0xC1087517 module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29975  ts: 0x155A147B6914CB4A OLD: ts: 0x155A147B6914E2BA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    46 ptr:  5001 data: 0xC10074F6 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29942  ts: 0x155A147B6954BA7A OLD: ts: 0x155A147B6954BAD4 dt: 0x000000000000005A tag: 2 type: 6
 ***            ADC data: block:    47 ptr:    35 data: 0xC108751C module:  4 fail: 0 range: 0 id:   264 channel:  8 adc: 29980  ts: 0x155A147B69B4958A OLD: ts: 0x155A147B69B4ACFA dt: 0x0000000000001770 tag: 3 type: 0
 ***            ADC data: block:    47 ptr:   129 data: 0xC10074F7 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29943  ts: 0x155A147B6A346C1A OLD: ts: 0x155A147B6A34708E dt: 0x0000000000000474 tag: 2 type: 6
 ***            ADC data: block:    47 ptr:   155 data: 0xC10074F9 module:  4 fail: 0 range: 0 id:   256 channel:  0 adc: 29945  ts: 0x155A147B6A54679A OLD: ts: 0x155A147B6A54698E dt: 0x00000000000001F4 tag: 2 type: 6
... 42869 more lines ...
  8   Thu Sep 20 09:39:17 2018 TDAIDA downstream window
The AIDA downstream window ('endcap' of snout) consists of 2x 50um aluminized mylar sheets
  7   Thu Sep 20 09:16:31 2018 TDImages of AIDA setup for DESPEC
Attachment 1 - detail of MSL type BB18(DS)-1000 p+n junction side strips, bias ring, multiple guard rings, filed 
plates and bond wires

Attachment 2 - MSL type BB18(DS)-1000 with Kapton PCBs attached

Attachment 3 - MSL type BB18(DS)-1000 with Kapton PCBs + 3M 1245 copper foil-screened Samtec FTSH ribbon cables 
attached

Attachment 4 - AIDA snout installed (with 1x DSSSD) with cabling to 4x FEE64s in AIDA support assembly

Attachment 5 - AIDA + FATIMA array

Attachment 6 - AIDA FEE64 PSUs, network switch, USB-controlled ac mains relay

Attachment 7 - AIDA MACBs, pulser, detector bias

Attachment 8 - FATIMA DAQ

Attachment 9 - DESPEC 19" racks on detector platform
Attachment 1: xIMG_1652.JPG
xIMG_1652.JPG
Attachment 2: xIMG_1653.JPG
xIMG_1653.JPG
Attachment 3: xIMG_1659.JPG
xIMG_1659.JPG
Attachment 4: xIMG_1660.JPG
xIMG_1660.JPG
Attachment 5: xIMG_1663.JPG
xIMG_1663.JPG
Attachment 6: xIMG_1664.JPG
xIMG_1664.JPG
Attachment 7: xIMG_1665.JPG
xIMG_1665.JPG
Attachment 8: xIMG_1666.JPG
xIMG_1666.JPG
Attachment 9: xIMG_1667.JPG
xIMG_1667.JPG
  6   Thu Sep 20 08:57:48 2018 OHThursday 20th September
09:57 Alpha run stoppped R2_0 -> R2_2

      Thresholds dropped from 0x64 to 0xa
      Pulser turned on again
      System wide checks ok
      FEE Temp ok - attachment 1
      Bias ok, leakage at 1.485uA at 160V
      Stats in similar place to yesterday - attachment 2

      Low Energy pulser walkthrough
      1.0V to 0.4V in 0.1V steps
      2 minutes per step
      Pulser at 50Hz
10:29 Run started R3_0

10:45 Run stopped R3_2
      1.8.L spectra for low energy pulser walkthrough - attachment 3

11:00
 High Energy pulser walkthrough
      1.8V to 0.V in 0.2V
      2 minutes per step
      Pulser at 50Hz

11:06 Run started R4_0

10:18 Run stopped R4_2
      1.8.L spectra for high energy pulser walkthrough - attachment 4
Attachment 1: 180920_0958_Temp.png
180920_0958_Temp.png
Attachment 2: 180920_1007_Stats.png
180920_1007_Stats.png
Attachment 3: 180920_1042_LowEPulser.png
180920_1042_LowEPulser.png
Attachment 4: 180920_1117_HighEPulser.png
180920_1117_HighEPulser.png
  5   Wed Sep 19 12:40:41 2018 OH, TDSingle DSSD Installation and Tests
On the 18th of September we installed DSSD 3208-14

Was found to have problems including exponential leakage behavior

  V      Ic
 -5   15.4uA
 -10  53uA
 -15  Tripped at over 100uA

Cables were double checked, bond wires were checked and no problems could be observed.

DSSD replaced with 3208-15

Bias at -100V Ic 10.2uA GP/grnd Jumpers removed,
                         rising upwards, possible short circuit as well.

Jumpers replaced

Bias -100V Ic 1.6uA
Bias -160V Ic 1.62uA

Possibility that mechanical sample 3208-17 and 3208-14 have been swapped. (To check at later date)

FEEs powered up and Adapter boards and all grounding installed with DSSD going to 9,10,11 and 12

Rates shown - attachment 1
Waveforms - attachment 2-4
1.8.l Spectra - attachment 5
FEEs 10 and 11 around 60FWHM and 9 and 12 around 80-90FWHM
Stats page - attachment 6

Aluminium foil used to make sure all ports are light tight shows a stats drop of around half - attachment 8

New peak widths

aida09 - 79.65
aida10 - 67.37
aida11 - 59.99
aida12 - 97.83

Going to begin a systematic study of shaping time

                              FWHM
Shaping Time   aida09   aida10   aida11   aida12
   
     8          79.65   67.37    59.99     97.83
     7          97.55   70.84    62.67     115.34
     6         100.72   70.04    62.77     120.05
     4         147.47   80.64    71.34     167.23
     2         217.62   114.98   133.35    232.47

Shaping time 8us 1.8.L Shots 
Shaping time 4us Stats appear obviously worse - attachment 7
                 1.8.L Widths - attachment 8
Shaping time 2us Stats appear worse again - attachment 9
                 1.8.L Widths - attachment 10
Shaping time 6us Stats are better than 2 and 4 - attachment 11
                 1.8.L widths - attachment 12
Shaping time 7us Stats are better than 6 - attachment 13
                 1.8.L widths - attachment 14


8us is the optimum shaping time.
Taken back to 8us and similar shaping times are observed.

Connected the ground of all 4 adapter boards with direct cables. Observed improvements in both FEEs 9 and 12
however the noise in 11 went
through the roof >500kHz - attachment 15

Tested just having 9,10 and 12 connected the rates are comparable to when disconnected - attachment 16

All 4 grounded together with a ground running to the snout as well stats No difference to all 4 together -
attachment 17
              1.8.L Widths - attachment 17
aida09 104.76
aida10 84.58
aida11 281.77
aida12 82.58

Tested just having 10 and 12 directly connected. Same side of the DSSD: Noise is the same as when disconnected.
Stats - attachment 19
1.8.L spectra - attachment 20

aida09 107.13
aida10 67.02
aida11 66.78
aida12 131.51

Tested having 11 and 12 connected directly. This is the two adapter boards responsible for the bias with 11
carrying the core and 12 the braid.
Massively reduced noise in 9 and 12, comparable noise in 10 but much more noise in 11. Stats - attachment 21
aida09 63.96
aida10 76.64
aida11 125.96
aida12 68.05
1.8.L spectra - attachment 23



18:39 Alpha run started in /TapeData/Sep18/R2
      Writing at 96kb/s
      11 Was fast at start bus asic control brought it down




                 
Attachment 1: 180919_1447_7us_1_8_L.png
180919_1447_7us_1_8_L.png
Attachment 2: 180919_1607_FEEsConnected.png
180919_1607_FEEsConnected.png
Attachment 3: 180919_1635_StatsSnoutGrnd.png
180919_1635_StatsSnoutGrnd.png
Attachment 4: 8_L_SnoutGround.png
8_L_SnoutGround.png
Attachment 5: 180919_1648_12and10Connect_Stats.png
180919_1648_12and10Connect_Stats.png
Attachment 6: 180919_1656_10and12_1_8_L.png
180919_1656_10and12_1_8_L.png
Attachment 7: 180919_1710_11and12_Stats.png
180919_1710_11and12_Stats.png
Attachment 8: 180919_1717_11and12_1_8_L.png
180919_1717_11and12_1_8_L.png
Attachment 9: 180919_1310_Rate.png
180919_1310_Rate.png
Attachment 10: 180919_1312_aida10W.png
180919_1312_aida10W.png
Attachment 11: 180919_1312_aida11W.png
180919_1312_aida11W.png
Attachment 12: 180919_1314_aida12W.png
180919_1314_aida12W.png
Attachment 13: 8_L.png
8_L.png
Attachment 14: 180919_1324_8usStats.png
180919_1324_8usStats.png
Attachment 15: 180919_1346_4usStats.png
180919_1346_4usStats.png
Attachment 16: 180919_1404_8us_1_8_L.png
180919_1404_8us_1_8_L.png
Attachment 17: 180919_1416_4us_1_8_L.png
180919_1416_4us_1_8_L.png
Attachment 18: 180919_1419_2usStats.png
180919_1419_2usStats.png
Attachment 19: 180919_1426_2us_1_8_L.png
180919_1426_2us_1_8_L.png
Attachment 20: 180919_1429_6usStats.png
180919_1429_6usStats.png
Attachment 21: 180919_1438_6us_1_8_L.png
180919_1438_6us_1_8_L.png
  4   Thu Sep 13 17:21:59 2018 TDAIDA upstream window
The upstream window of the AIDA detector assembly consists of 2x 50um aluminized mylar
  3   Thu Sep 6 15:42:53 2018 OH, TDFEE64 system console logs
Kernel panics during boot cab be observed in files ttyUSB1 and ttyUSB5

In each case the FEE64 successfully completed reboot c. 5 minutes later
Attachment 1: logs.tar
  2   Thu Sep 6 15:25:19 2018 OH, TDAIDA@DESPEC Setup Photos
  1   Thu Sep 6 15:11:17 2018 OH, TDTuesday 4 September - Thursday 6 September
Attachment 1 - FEE64 temperatures illustrating (we assume) a faulty PSU temperature sensor for aida07

Attachment 2 - FEE64 temperatures following replacement of aida07

Attachment 3-5 - 1.*.W waveforms for aida01 - aida03 - LED settings for aida02 incorrect, aida01 adaptor PCB 
incorrectly aligned

Attachment 6-7 - 1.*.W waveforms for aida02 and aida04

Attachment 8-9 - aida01 - aida04 1.8.L spectra full range and expanded scale
                          pulser peak width c. 15-20 channels FWHM

Attachment 10-12 - good events stats for all FEE64s and all statistics for aida01 and aida05
                   Note aida01 - aida04 show rates expected, all other FEE64s show very rates due to noise
                   aida01 - aida04 adaptor PCB are well grounded to FEE64 copper mezzanine/cooling plate, the other 
FEE64s are not

Attachment 13-14 - aida01 - aida04 rate spectra before and after correcting alignment of aida01 adaptor PCB

Attachment 15 - aida01 - aida04 1.8.L spectra, expanded scale

Attachment 16 - illustrates high frequency noise (period 5 samples = 100ns) for aida01 which is not observed for 
aida02 - aida03
                subsequently determined that this is not due to a possible adaptor PCB fault by changing the adaptor 
PCB
                Note that with bias turned on this noise did appear when Bias was on but all the other groudning
connectors for aida05-aida12
were not in effect.

Attachment 17-19 - PuTTY connection to CAEN 1419ET 4ch high voltage supply illustrating PuTTY, N1419ET configuration 
and leakage current with 100V applied to adaptor PCBs (no input cabling or DSSSD connected)

Note the CAEN N1419ET appears as a 'USB ACM' device and we connect using /dev/ttyACM<n>. 
Attachment 1: 180904_1615_FeeTemp.png
180904_1615_FeeTemp.png
Attachment 2: 180905_1444_FEETemp.png
180905_1444_FEETemp.png
Attachment 3: 180905_1636_aida01W.png
180905_1636_aida01W.png
Attachment 4: 180905_1643_aida02W.png
180905_1643_aida02W.png
Attachment 5: 180905_1644_aida03W.png
180905_1644_aida03W.png
Attachment 6: 180905_1646_aida02W.png
180905_1646_aida02W.png
Attachment 7: 180905_1647_aida04W.png
180905_1647_aida04W.png
Attachment 8: 8_L.png
8_L.png
Attachment 9: 8_L_Zoom.png
8_L_Zoom.png
Attachment 10: 180905_1653_Stats.png
180905_1653_Stats.png
Attachment 11: 180905_1654_aida01Stats.png
180905_1654_aida01Stats.png
Attachment 12: 180905_1654_aida05Stats.png
180905_1654_aida05Stats.png
Attachment 13: 180905_1657_HitRateAida1-4.png
180905_1657_HitRateAida1-4.png
Attachment 14: 180905_1703_HitRateZoomPostAlign.png
180905_1703_HitRateZoomPostAlign.png
Attachment 15: 180905_1707_aida01_1_8_L.png
180905_1707_aida01_1_8_L.png
Attachment 16: 180905_1709_aida01W.png
180905_1709_aida01W.png
Attachment 17: 180906_1424_biasWindow.png
180906_1424_biasWindow.png
Attachment 18: 180906_1425_Putty.png
180906_1425_Putty.png
Attachment 19: 180906_1441_BiasOnNoDSSD.png
180906_1441_BiasOnNoDSSD.png
ELOG V3.1.4-unknown